Stream scanner for identifying signature matches

ABSTRACT

System and methods implemented in a node in a cloud-based security system include obtaining a plurality of rules each define via a rule syntax that includes a rule header and rule options, wherein each rule header is used to for a rule database lookup, and each rule options is used to specify details about the associated rule; monitoring data associated with a user of the cloud-based security system; analyzing the data with the plurality of rules; and performing one or more security functions on the data based on triggering of a rule of the plurality of rules.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present disclosure is a continuation-in-part of U.S. patentapplication Ser. No. 16/781,505, filed Feb. 4, 2020, and entitled“Multi-tenant cloud-based firewall systems and methods,” which is acontinuation of U.S. patent application Ser. No. 14/943,579, filed Nov.17, 2015 (now U.S. Pat. No. 10,594,656, issued Mar. 17, 2020), andentitled “Multi-tenant cloud-based firewall systems and methods,” thecontents of each incorporated by reference in their entirety.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to computer networking systemsand methods. More particularly, the present disclosure relates to astream scanner for identifying signature matches.

BACKGROUND OF THE DISCLOSURE

Conventionally, Intrusion Prevention Systems (IPS), also known asIntrusion Detection and Prevention Systems (IDPS), are network securityappliances that monitor network or system activities for maliciousactivity. The main functions of an IPS are to identify maliciousactivity, log information about this activity, report it, and attempt toblock or stop it. Intrusion prevention systems are considered extensionsof Intrusion Detection Systems (IDS) because they both monitor networktraffic and/or system activities for malicious activity. The maindifferences are, unlike IDS, IPS systems are placed in-line and are ableto actively prevent or block intrusions that are detected. An IPS systemcan take such actions as sending an alarm, dropping detected maliciouspackets, resetting a connection or blocking traffic from the offendingInternet Protocol (IP) address. An IPS also can correct CyclicRedundancy Check (CRC) errors, defragment packet streams, mitigateTransmission Control Protocol (TCP) sequencing issues, and clean upunwanted transport and network layer options.

Conventional IPS systems are physical devices and can be network-based,wireless, behavioral, or host-based. A network-based IPS can monitortraffic for a specific network. A wireless IPS can physically becollocated with a wireless network to monitor and analyze wirelessnetwork protocols. A behavioral system can examine network traffic toidentify threats that generate unusual traffic flows such as DistributedDenial of Service (DDoS) attacks, etc. Finally, a host-based system isexecuted on a single host, i.e., a host-based system monitors a singlehost to identify suspicious activity associated with the host.

Information Technology (IT) is moving away from physical appliances;network perimeters are disappearing with user's mobile devices, 5Gspeeds, Bring Your Own Device (BYOD), etc. As such, physical IPSappliances are not able to capture and protect against threats wherethere is no perimeter. Enterprise users and applications (“apps”) haveleft the enterprise network, but conventional IPS systems remain sittingin the data center. Mobility and cloud migration are causing the IPSinvestment, and the associated security, to run blind. Conventional IPSwas designed to protect servers sitting in the data center, butintrusions now leverage the weakest link: the user. Enterprises cannotafford Inspection compromises and Secure Sockets Layer (SSL)limitations. The demands of inspecting all traffic, including SSL—wheremost threats hide—has been a challenge for conventional IPS approaches.

Also, in networks, firewalls monitor and control incoming and outgoingnetwork traffic based on predetermined security rules. A firewalltypically establishes a barrier between a trusted, secure internalnetwork and another outside network, such as the Internet, that isassumed not to be secure or trusted. Firewalls are often categorized aseither network firewalls or host-based firewalls. Network firewalls area software appliance running on general-purpose hardware orhardware-based firewall computer appliances that filter traffic betweentwo or more networks. Host-based firewalls provide a layer of softwareon one host that controls network traffic in and out of that singlemachine. Firewall appliances may also offer other functionality to theinternal network they protect, such as acting as a Dynamic HostConfiguration Protocol (DHCP) or Virtual Private Network (VPN) serverfor that network. Disadvantageously, conventional firewalls, eithernetwork firewalls or host-based firewalls are physical devices locatedat the boundary between the internal network and the outside network(the Internet). That is, network firewalls are appliance-based at thenetwork boundary, and host-based firewalls are on a single device. Thisscheme does not reflect the evolving network of cloud-basedconnectivity, Bring Your Own Device (BYOD), etc. For example, a roadwarrior, home user, or employee with their mobile device does not havethe benefit of a network firewall outside of the internal network. Also,mobile devices and their associated operating systems may not allowhost-based firewalls. Thus, there is a need for next-generation firewallsystems and methods that can adapt to the evolving network.

BRIEF SUMMARY OF THE DISCLOSURE

The present disclosure relates to a stream scanner for identifyingsignature matches. Specifically, the present disclosure utilizes aSnort-like format for writing signatures. The present disclosure is usedto analyze data traffic to identify stream-based filters patterns andmatch rules based on a Snort style rule syntax. Such stream scanning isused in a cloud-based system for a cloud-based Intrusion PreventionSystem (IPS), a cloud-based firewall, real-time traffic analysis andpacket logging, and the like. Also, the present disclosure relates to acloud-based Intrusion Prevention System (IPS). A cloud-based IPS enablesIPS threat protection where traditional IPS systems cannot, namely, thecloud-based IPS follows users, no matter the connection type, location,device type, operating system, etc. Enterprise IT has always-on threatprotection and visibility. The cloud-based IPS works across a full suiteof technologies such as firewall, sandbox, Cloud Access Security Broker(CASB), Data Leakage Prevention (DLP), etc. to stop various types ofattacks. The cloud-based IPS provides threat protection from botnets,advanced threats, and zero-day vulnerabilities, along with contextualinformation about the user, app, and threat. The cloud-based IPS isdelivered as a cloud-based service, so inspection demands scaleautomatically, updates are immediate, and the need to manage hardware isremoved.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated and described herein withreference to the various drawings, in which like reference numbers areused to denote like system components/method steps, as appropriate, andin which:

FIG. 1 is a network diagram of a distributed security system;

FIG. 2 is a network diagram of the distributed security system of FIG. 1illustrating various components in more detail;

FIG. 3 is a block diagram of a server which may be used in thedistributed security system of FIG. 1 or with any other cloud-basedsystem;

FIG. 4 is a block diagram of a mobile device which may be used in thesystem of FIG. 1 or with any other cloud-based system;

FIG. 5 is a network diagram of a generalized cloud-based system;

FIG. 6 is a network diagram of a network with a distributed securitycloud providing DNS augmented security;

FIG. 7 is a network diagram of a network with a firewall in accordancewith the multi-tenant cloud-based firewall systems and methods;

FIG. 8 is a network diagram of a network illustrating example use casesof the firewall;

FIG. 9 is a screenshot associated with the firewall illustrating examplenetwork services;

FIG. 10 is a screenshot associated with the firewall illustrates exampleapplications;

FIG. 11 is a diagram of a Deep Packet Inspection (DPI) engine for thefirewall;

FIG. 12A is a screenshot of defining a firewall filtering rule;

FIG. 12B is another screenshot of defining a firewall filtering rule;

FIG. 13 is screenshots of editing IP groups;

FIG. 14 is screenshots of editing a network service;

FIG. 15 is a flow diagram that illustrates packet flow through the cloudnode;

FIG. 16 is a flowchart of a process for packet flow through the firewallfrom a client;

FIG. 17 is a flowchart of a process for packet flow through the firewallfrom a server;

FIG. 18 is a screenshot of creating firewall policies;

FIG. 19 is a screenshot of a NAT configuration;

FIG. 20 is a screenshot of a user authentication screen;

FIG. 21 is a screenshot of DNS policy;

FIG. 22 is a screenshot of a reporting screen for firewall insights;

FIG. 23 is a screenshot of an interactive report for firewall insights;

FIG. 24 is a screenshot of a graph of usage trends through the firewall;

FIG. 25 is graphs of top firewall protocols in sessions and bytes;

FIGS. 26 and 27 are network diagrams illustrating deployment modes ofthe cloud firewall;

FIG. 28 is a block diagram of functionality in the processing node orthe cloud node for implementing various functions described herein

FIG. 29 is a block diagram and flowchart of how a packet is processedinside the processing node or the cloud node;

FIG. 30 is a block diagram of a cloud IPS system, implemented via thecloud system of FIG. 5 and/or the distributed security system of FIG. 1;

FIG. 31 is a diagram of detection filters and event filters usedtogether for a stream scanner;

FIG. 32 is a diagram illustrating rule grouping in a lookup tree;

FIG. 33 is a diagram of an example rule option Directed Acyclic Graph(DAG);

FIG. 34 is a diagram of the overall flow when data arrives on the streamscanning engine;

FIG. 35 is a flowchart of scan processing at a cloud node; and

FIG. 36 is a flow diagram of functions performed by the cloud nodebetween a firewall module and a proxy module.

DETAILED DESCRIPTION OF THE DISCLOSURE

Again, the present disclosure relates to a stream scanner foridentifying signature matches. Specifically, the present disclosureutilizes a Snort-like format for writing signatures. The presentdisclosure is used to analyze data traffic to identify stream-basedfilters patterns and match rules based on a Snort style rule syntax.Such stream scanning is used in a cloud-based system for a cloud-basedIntrusion Prevention System (IPS), a cloud-based firewall, real-timetraffic analysis and packet logging, and the like. Also, the presentdisclosure relates to a cloud-based Intrusion Prevention System (IPS). Acloud-based IPS enables IPS threat protection where traditional IPSsystems cannot, namely, the cloud-based IPS follows users, no matter theconnection type, location, device type, operating system, etc.Enterprise IT has always-on threat protection and visibility. Thecloud-based IPS works across a full suite of technologies such asfirewall, sandbox, Cloud Access Security Broker (CASB), Data LeakagePrevention (DLP), etc. to stop various types of attacks. The cloud-basedIPS provides threat protection from botnets, advanced threats, andzero-day vulnerabilities, along with contextual information about theuser, app, and threat. The cloud-based IPS is delivered as a cloud-basedservice, so inspection demands scale automatically, updates areimmediate, and the need to manage hardware is removed.

Also, the present disclosure relates to a multi-tenant cloud-basedfirewall. The firewall systems and methods can operate overlaid withexisting branch office firewalls or routers as well as eliminate theneed for physical firewalls. The firewall systems and methods canprotect users at user level control, regardless of location, device,etc., over all ports and protocols (not only ports 80/443) whileproviding administrators a single unified policy for Internet access andintegrated reporting and visibility. In an embodiment, the firewallsystems and methods can eliminate dedicated hardware at user locations(e.g., branch or regional offices, etc.), providing a software-basedcloud solution, such as a Virtualized Network Function (VNF) in thecloud. The firewall systems and methods support application awareness toidentify application regardless of port, protocol, evasive tactic, orSecure Sockets Layer (SSL); user awareness to identify users, groups,and locations regardless of physical Internet Protocol (IP) address;visibility and policy management providing globally unifiedadministration, policy management, and reporting; threat protection andcompliance to block threats and data leaks in real-time; highperformance through an in-line cloud-based, scalable system; and costeffectiveness with rapid deployment. In an embodiment, the firewallsystems and methods are described implemented through or in conjunctionwith a distributed, cloud-based security system and the firewall systemsand methods can be integrated with sandboxing, web security, DataLeakage Prevention (DLP), content filtering, SSL inspection, malwareprotection, and cloud-scale correlation, anti-virus, bandwidthmanagement reporting and analytics, and the like.

§ 1.0 Example High-Level System Architecture—Cloud-Based Security System

FIG. 1 is a block diagram of a distributed security system 100. Thesystem 100 may, for example, be implemented as an overlay network in awide area network (WAN), such as the Internet, a local area network(LAN), or the like. The system 100 includes processing nodes (PN) 110,that proactively detect and preclude the distribution of securitythreats, e.g., malware, spyware, viruses, email spam, DLP, contentfiltering, etc., and other undesirable content sent from or requested byan external system. The processing nodes 110 can also log activity andenforce policies, including logging changes to the various componentsand settings in the system 100. Example external systems may include anenterprise 200, a computer device 220, and a mobile device 230, or othernetwork and computing systems communicatively coupled to the system 100.In an embodiment, each of the processing nodes 110 may include adecision system, e.g., data inspection engines that operate on a contentitem, e.g., a web page, a file, an email message, or some other data ordata communication that is sent from or requested by one of the externalsystems. In an embodiment, all data destined for or received from theInternet is processed through one of the processing nodes 110. Inanother embodiment, specific data specified by each external system,e.g., only email, only executable files, etc., is processed through oneof the processing node 110.

Each of the processing nodes 110 may generate a decision vector D=[d1,d2, . . . , dn] for a content item of one or more parts C=[c1, c2, . . ., cm]. Each decision vector may identify a threat classification, e.g.,clean, spyware, malware, undesirable content, innocuous, spam email,unknown, etc. For example, the output of each element of the decisionvector D may be based on the output of one or more data inspectionengines. In an embodiment, the threat classification may be reduced to asubset of categories, e.g., violating, non-violating, neutral, unknown.Based on the subset classification, the processing node 110 may allowthe distribution of the content item, preclude distribution of thecontent item, allow distribution of the content item after a cleaningprocess, or perform threat detection on the content item. In anembodiment, the actions taken by one of the processing nodes 110 may bedetermined by the threat classification of the content item and on asecurity policy of the external system to which the content item isbeing sent from or from which the content item is being requested by. Acontent item is violating if, for any part C=[c1, c2, . . . , cm] of thecontent item, at any of the processing nodes 110, any one of the datainspection engines generates an output that results in a classificationof “violating.”

Each of the processing nodes 110 may be implemented by one or more ofcomputer and communications devices, e.g., server computers, gateways,switches, etc., such as the server 300 described in FIG. 3. In anembodiment, the processing nodes 110 may serve as an access layer 150.The access layer 150 may, for example, provide external system access tothe security system 100. In an embodiment, each of the processing nodes110 may include Internet gateways and one or more servers, and theprocessing nodes 110 may be distributed through a geographic region,e.g., throughout a country, region, campus, etc. According to a serviceagreement between a provider of the system 100 and an owner of anexternal system, the system 100 may thus provide security protection tothe external system at any location throughout the geographic region.

Data communications may be monitored by the system 100 in a variety ofways, depending on the size and data requirements of the externalsystem. For example, an enterprise 200 may have multiple routers,switches, etc. that are used to communicate over the Internet, and therouters, switches, etc. may be configured to establish communicationsthrough the nearest (in traffic communication time, for example)processing node 110. A mobile device 230 may be configured tocommunicate to a nearest processing node 110 through any availablewireless access device, such as an access point, or a cellular gateway.A single computer device 220, such as a consumer's personal computer,may have its browser and email program configured to access the nearestprocessing node 110, which, in turn, serves as a proxy for the computerdevice 220. Alternatively, an Internet provider may have all of itscustomer traffic processed through the processing nodes 110.

In an embodiment, the processing nodes 110 may communicate with one ormore authority nodes (AN) 120. The authority nodes 120 may store policydata for each external system and may distribute the policy data to eachof the processing nodes 110. The policy may, for example, definesecurity policies for a protected system, e.g., security policies forthe enterprise 200. Example policy data may define access privileges forusers, websites and/or content that is disallowed, restricted domains,etc. The authority nodes 120 may distribute the policy data to theaccess nodes 110. In an embodiment, the authority nodes 120 may alsodistribute threat data that includes the classifications of contentitems according to threat classifications, e.g., a list of knownviruses, a list of known malware sites, spam email domains, a list ofknown phishing sites, etc. The distribution of threat data between theprocessing nodes 110 and the authority nodes 120 may be implemented bypush and pull distribution schemes described in more detail below. In anembodiment, each of the authority nodes 120 may be implemented by one ormore computer and communication devices, e.g., server computers,gateways, switches, etc., such as the server 300 described in FIG. 3. Insome embodiments, the authority nodes 120 may serve as an applicationlayer 170. The application layer 170 may, for example, manage andprovide policy data, threat data, and data inspection engines anddictionaries for the processing nodes 110.

Other application layer functions may also be provided in theapplication layer 170, such as a user interface (UI) front-end 130. Theuser interface front-end 130 may provide a user interface through whichusers of the external systems may provide and define security policies,e.g., whether email traffic is to be monitored, whether certain websites are to be precluded, etc. Another application capability that maybe provided through the user interface front-end 130 is securityanalysis and log reporting. The underlying data on which the securityanalysis and log reporting functions operate are stored in logging nodes(LN) 140, which serve as a data logging layer 170. Each of the loggingnodes 140 may store data related to security operations and networktraffic processed by the processing nodes 110 for each external system.In an embodiment, the logging node 140 data may be anonymized so thatdata identifying an enterprise is removed or obfuscated. For example,identifying data may be removed to provide an overall system summary ofsecurity processing for all enterprises and users without revealing theidentity of any one account. Alternatively, identifying data may beobfuscated, e.g., provide a random account number each time it isaccessed, so that an overall system summary of security processing forall enterprises and users may be broken out by accounts withoutrevealing the identity of any one account. In another embodiment, theidentifying data and/or logging node 140 data may be further encrypted,e.g., so that only the enterprise (or user if a single user account) mayhave access to the logging node 140 data for its account. Otherprocesses of anonymizing, obfuscating, or securing logging node 140 datamay also be used. Note, as described herein, the systems and methods fortracking and auditing changes in a multi-tenant cloud system can beimplemented in the data logging layer 160, for example.

In an embodiment, an access agent 180 may be included in the externalsystems. For example, the access agent 180 is deployed in the enterprise200. The access agent 180 may, for example, facilitate securityprocessing by providing a hash index of files on a client device to oneof the processing nodes 110, or may facilitate authentication functionswith one of the processing nodes 110, e.g., by assigning tokens forpasswords and sending only the tokens to a processing node so thattransmission of passwords beyond the network edge of the enterprise isminimized. Other functions and processes may also be facilitated by theaccess agent 180. In an embodiment, the processing node 110 may act as aforward proxy that receives user requests to external servers addresseddirectly to the processing node 110. In another embodiment, theprocessing node 110 may access user requests that are passed through theprocessing node 110 in a transparent mode. A protected system, e.g.,enterprise 200, may, for example, choose one or both of these modes. Forexample, a browser may be configured either manually or through theaccess agent 180 to access the processing node 110 in a forward proxymode. In the forward proxy mode, all accesses are addressed to theprocessing node 110.

In an embodiment, an enterprise gateway may be configured so that userrequests are routed through the processing node 110 by establishing acommunication tunnel between the enterprise gateway and the processingnode 110. For establishing the tunnel, existing protocols such asgeneric routing encapsulation (GRE), layer two tunneling protocol(L2TP), or other Internet Protocol (IP) security protocols may be used.In another embodiment, the processing nodes 110 may be deployed atInternet service provider (ISP) nodes. The ISP nodes may redirectsubject traffic to the processing nodes 110 in a transparent proxy mode.Protected systems, such as the enterprise 200, may use a multiprotocollabel switching (MPLS) class of service for indicating the subjecttraffic that is to be redirected. For example, at or within theenterprise, the access agent 180 may be configured to perform MPLSlabeling. In another transparent proxy mode embodiment, a protectedsystem, such as the enterprise 200, may identify the processing node 110as a next hop router for communication with the external servers.

Generally, the distributed security system 100 may generally refer to anexample cloud-based security system. Other cloud-based security systemsand generalized cloud-based systems are contemplated for the systems andmethods for tracking and auditing changes in a multi-tenant cloudsystem. Cloud computing systems and methods abstract away physicalservers, storage, networking, etc. and instead offer these as on-demandand elastic resources. The National Institute of Standards andTechnology (NIST) provides a concise and specific definition whichstates cloud computing is a model for enabling convenient, on-demandnetwork access to a shared pool of configurable computing resources(e.g., networks, servers, storage, applications, and services) that canbe rapidly provisioned and released with minimal management effort orservice provider interaction. Cloud computing differs from the classicclient-server model by providing applications from a server that areexecuted and managed by a client's web browser, with no installed clientversion of an application required. Centralization gives cloud serviceproviders complete control over the versions of the browser-basedapplications provided to clients, which removes the need for versionupgrades or license management on individual client computing devices.The phrase “software as a service” (SaaS) is sometimes used to describeapplication programs offered through cloud computing. A common shorthandfor a provided cloud computing service (or even an aggregation of allexisting cloud services) is “the cloud.” The distributed security system100 is illustrated herein as one example embodiment of a cloud-basedsystem, and those of ordinary skill in the art will recognize thetracking and auditing systems and methods contemplate operation on anycloud-based system.

§ 2.0 Example Detailed System Architecture and Operation

FIG. 2 is a block diagram of various components of the distributedsecurity system 100 in more detail. Although FIG. 2 illustrates only onerepresentative component processing node 110, authority node 120, andlogging node 140, those of ordinary skill in the art will appreciatethere may be many of each of the component nodes 110, 120, and 140present in the system 100. A wide area network (WAN) 101, such as theInternet, or some other combination of wired and/or wireless networks,communicatively couples the processing node 110, the authority node 120,and the logging node 140 to one another. The external systems 200, 220,and 230 likewise communicate over the WAN 101 with each other or otherdata providers and publishers. Some or all of the data communication ofeach of the external systems 200, 220 and 230 may be processed throughthe processing node 110.

FIG. 2 also shows the enterprise 200 in more detail. The enterprise 200may, for example, include a firewall (FW) 202 protecting an internalnetwork that may include one or more enterprise servers 216, alightweight directory access protocol (LDAP) server 212, and other dataor data stores 214. Another firewall 203 may protect an enterprisesubnet that can include user computers 206 and 208 (e.g., laptop anddesktop computers). The enterprise 200 may communicate with the WAN 101through one or more network devices, such as a router, gateway, switch,etc. The LDAP server 212 may store, for example, user login credentialsfor registered users of the enterprise 200 system. Such credentials mayinclude user identifiers, login passwords, and a login historyassociated with each user identifier. The other data stores 214 mayinclude sensitive information, such as bank records, medical records,trade secret information, or any other information warranting protectionby one or more security measures.

In an embodiment, a client access agent 180 a may be included on aclient computer 208. The client access agent 180 a may, for example,facilitate security processing by providing a hash index of files on theuser computer 208 to a processing node 110 for malware, virus detection,etc. Other security operations may also be facilitated by the accessagent 180 a. In another embodiment, a server access agent 180 mayfacilitate authentication functions with the processing node 110, e.g.,by assigning tokens for passwords and sending only the tokens to theprocessing node 110 so that transmission of passwords beyond the networkedge of the enterprise 200 is minimized. Other functions and processesmay also be facilitated by the server access agent 180 b. The computerdevice 220 and the mobile device 230 may also store informationwarranting security measures, such as personal bank records, medicalinformation, and login information, e.g., login information to theserver 206 of the enterprise 200, or to some other secured data providerserver. The computer device 220 and the mobile device 230 can also storeinformation warranting security measures, such as personal bank records,medical information, and login information, e.g., login information to aserver 216 of the enterprise 200, or to some other secured data providerserver.

§ 2.1 Example Processing Node Architecture

In an embodiment, the processing nodes 110 are external to network edgesof the external systems 200, 220, and 230. Each of the processing nodes110 stores security policies 113 received from the authority node 120and monitors content items requested by or sent from the externalsystems 200, 220, and 230. In an embodiment, each of the processingnodes 110 may also store a detection process filter 112 and/or threatdata 114 to facilitate the decision of whether a content item should beprocessed for threat detection. A processing node manager 118 may manageeach content item in accordance with the security policy data 113, andthe detection process filter 112 and/or threat data 114, if stored atthe processing node 110, so that security policies for a plurality ofexternal systems in data communication with the processing node 110 areimplemented external to the network edges for each of the externalsystems 200, 220 and 230. For example, depending on the classificationresulting from the monitoring, the content item may be allowed,precluded, or threat detected. In general, content items that arealready classified as “clean” or not posing a threat can be allowed,while those classified as “violating” may be precluded. Those contentitems having an unknown status, e.g., content items that have not beenprocessed by the system 100, may be threat detected to classify thecontent item according to threat classifications.

The processing node 110 may include a state manager 116A. The statemanager 116A may be used to maintain the authentication and theauthorization states of users that submit requests to the processingnode 110. Maintenance of the states through the state manager 116A mayminimize the number of authentication and authorization transactionsthat are necessary to process a request. The processing node 110 mayalso include an epoch processor 116B. The epoch processor 116B may beused to analyze authentication data that originated at the authoritynode 120. The epoch processor 116B may use an epoch ID to validatefurther the authenticity of authentication data. The processing node 110may further include a source processor 116C. The source processor 116Cmay be used to verify the source of authorization and authenticationdata. The source processor 116C may identify improperly obtainedauthorization and authentication data, enhancing the security of thenetwork. Collectively, the state manager 116A, the epoch processor 116B,and the source processor 116C operate as data inspection engines.

Because the amount of data being processed by the processing nodes 110may be substantial, the detection processing filter 112 may be used asthe first stage of an information lookup procedure. For example, thedetection processing filter 112 may be used as a front end to a lookingof the threat data 114. Content items may be mapped to index values ofthe detection processing filter 112 by a hash function that operates onan information key derived from the information item. The informationkey is hashed to generate an index value (i.e., a bit position). A valueof zero in a bit position in the guard table can indicate, for example,the absence of information, while a one in that bit position canindicate the presence of information. Alternatively, a one could be usedto represent absence, and a zero to represent presence. Each contentitem may have an information key that is hashed. For example, theprocessing node manager 118 may identify the Uniform Resource Locator(URL) address of URL requests as the information key and hash the URLaddress; or may identify the file name and the file size of anexecutable file information key and hash the file name and file size ofthe executable file. Hashing an information key to generate an index andchecking a bit value at the index in the detection processing filter 112generally requires less processing time than actually searching threatdata 114. The use of the detection processing filter 112 may improve thefailure query (i.e., responding to a request for absent information)performance of database queries and/or any general information queries.Because data structures are generally optimized to access informationthat is present in the structures, failure query performance has agreater effect on the time required to process information searches forvery rarely occurring items, e.g., the presence of file information in avirus scan log or a cache where many or most of the files transferred ina network have not been scanned or cached. Using the detectionprocessing filter 112, however, the worst-case additional cost is onlyon the order of one, and thus its use for most failure queries saves onthe order of m log m, where m is the number of information recordspresent in the threat data 114.

The detection processing filter 112 thus improves the performance ofqueries where the answer to a request for information is usuallypositive. Such instances may include, for example, whether a given filehas been virus scanned, whether the content at a given URL has beenscanned for inappropriate (e.g., pornographic) content, whether a givenfingerprint matches any of a set of stored documents, and whether achecksum corresponds to any of a set of stored documents. Thus, if thedetection processing filter 112 indicates that the content item has notbeen processed, then a worst-case null lookup operation into the threatdata 114 is avoided, and threat detection can be implementedimmediately. The detection processing filter 112 thus complements thethreat data 114 that capture positive information. In an embodiment, thedetection processing filter 112 may be a Bloom filter implemented by asingle hash function. The Bloom filter may be sparse table, i.e., thetables include many zeros and few ones, and the hash function is chosento minimize or eliminate false negatives which are, for example,instances where an information key is hashed to a bit position, and thatbit position indicates that the requested information is absent when itis actually present.

§ 2.2 Example Authority Node Architecture

In general, the authority node 120 includes a data store that storesmaster security policy data 123 for each of the external systems 200,220, and 230. An authority node manager 128 may be used to manage themaster security policy data 123, e.g., receive input from users of eachof the external systems defining different security policies, and maydistribute the master security policy data 123 to each of the processingnodes 110. The processing nodes 110 then store a local copy of thesecurity policy data 113. The authority node 120 may also store a masterdetection process filter 122. The detection processing filter 122 mayinclude data indicating whether content items have been processed by oneor more of the data inspection engines 116 in any of the processingnodes 110. The authority node manager 128 may be used to manage themaster detection processing filter 122, e.g., receive updates fromprocessing nodes 110 when the processing node 110 has processed acontent item and update the master detection processing filter 122. Forexample, the master detection processing filter 122 may be distributedto the processing nodes 110, which then stores a local copy of thedetection processing filter 112.

In an embodiment, the authority node 120 may include an epoch manager126. The epoch manager 126 may be used to generate authentication dataassociated with an epoch ID. The epoch ID of the authentication data isa verifiable attribute of the authentication data that can be used toidentify fraudulently created authentication data. In an embodiment, thedetection processing filter 122 may be a guard table. The processingnode 110 may, for example, use the information in the local detectionprocessing filter 112 to quickly determine the presence and/or absenceof information, e.g., whether a particular URL has been checked formalware; whether a particular executable has been virus scanned, etc.The authority node 120 may also store master threat data 124. The masterthreat data 124 may classify content items by threat classifications,e.g., a list of known viruses, a list of known malware sites, spam emaildomains, a list of known or detected phishing sites, etc. The authoritynode manager 128 may be used to manage the master threat data 124, e.g.,receive updates from the processing nodes 110 when one of the processingnodes 110 has processed a content item and update the master threat data124 with any pertinent results. In some implementations, the masterthreat data 124 may be distributed to the processing nodes 110, whichthen store a local copy of the threat data 114. In another embodiment,the authority node 120 may also monitor the health of each of theprocessing nodes 110, e.g., the resource availability in each of theprocessing nodes 110, detection of link failures, etc. Based on theobserved health of each of the processing nodes 110, the authority node120 may redirect traffic among the processing nodes 110 and/or balancetraffic among the processing nodes 110. Other remedial actions andprocesses may also be facilitated by the authority node 110.

§ 2.3 Example Processing Node and Authority Node Communications

The processing node 110 and the authority node 120 may be configuredaccording to one or more push and pull processes to manage content itemsaccording to security policy data 113 and/or 123, detection processfilters 112 and/or 122, and the threat data 114 and/or 124. In a threatdata push implementation, each of the processing nodes 110 stores policydata 113 and threat data 114. The processing node manager 118 determineswhether a content item requested by or transmitted from an externalsystem is classified by the threat data 114. If the content item isdetermined to be classified by the threat data 114, then the processingnode manager 118 may manage the content item according to the securityclassification of the content item and the security policy of theexternal system. If, however, the content item is determined not to beclassified by the threat data 114, then the processing node manager 118may cause one or more of the data inspection engines 117 to perform thethreat detection processes to classify the content item according to athreat classification. Once the content item is classified, theprocessing node manager 118 generates a threat data update that includesdata indicating the threat classification for the content item from thethreat detection process and transmits the threat data update to anauthority node 120.

The authority node manager 128, in response to receiving the threat dataupdate, updates the master threat data 124 stored in the authority nodedata store according to the threat data update received from theprocessing node 110. In an embodiment, the authority node manager 128may automatically transmit the updated threat data to the otherprocessing nodes 110. Accordingly, threat data for new threats as thenew threats are encountered are automatically distributed to eachprocessing node 110. Upon receiving the new threat data from theauthority node 120, each of processing node managers 118 may store theupdated threat data in the locally stored threat data 114.

In a threat data pull and push implementation, each of the processingnodes 110 stores policy data 113 and threat data 114. The processingnode manager 118 determines whether a content item requested by ortransmitted from an external system is classified by the threat data114. If the content item is determined to be classified by the threatdata 114, then the processing node manager 118 may manage the contentitem according to the security classification of the content item andthe security policy of the external system. If, however, the contentitem is determined not to be classified by the threat data, then theprocessing node manager 118 may request responsive threat data for thecontent item from the authority node 120. Because processing a contentitem may consume valuable resource and time, in some implementations,the processing node 110 may first check with the authority node 120 forthreat data 114 before committing such processing resources.

The authority node manager 128 may receive the responsive threat datarequest from the processing node 110 and may determine if the responsivethreat data is stored in the authority node data store. If responsivethreat data is stored in the master threat data 124, then the authoritynode manager 128 provide a reply that includes the responsive threatdata to the processing node 110 so that the processing node manager 118may manage the content item in accordance with the security policy data112 and the classification of the content item. Conversely, if theauthority node manager 128 determines that responsive threat data is notstored in the master threat data 124, then the authority node manager128 may provide a reply that does not include the responsive threat datato the processing node 110. In response, the processing node manager 118can cause one or more of the data inspection engines 116 to perform thethreat detection processes to classify the content item according to athreat classification. Once the content item is classified, theprocessing node manager 118 generates a threat data update that includesdata indicating the threat classification for the content item from thethreat detection process, and transmits the threat data update to anauthority node 120. The authority node manager 128 can then update themaster threat data 124. Thereafter, any future requests related toresponsive threat data for the content item from other processing nodes110 can be readily served with responsive threat data.

In a detection process filter and threat data push implementation, eachof the processing nodes 110 stores a detection process filter 112,policy data 113, and threat data 114. The processing node manager 118accesses the detection process filter 112 to determine whether thecontent item has been processed. If the processing node manager 118determines that the content item has been processed, it may determine ifthe content item is classified by the threat data 114. Because thedetection process filter 112 has the potential for a false positive, alookup in the threat data 114 may be implemented to ensure that a falsepositive has not occurred. The initial check of the detection processfilter 112, however, may eliminate many null queries to the threat data114, which, in turn, conserves system resources and increasesefficiency. If the content item is classified by the threat data 114,then the processing node manager 118 may manage the content item inaccordance with the security policy data 113 and the classification ofthe content item. Conversely, if the processing node manager 118determines that the content item is not classified by the threat data114, or if the processing node manager 118 initially determines throughthe detection process filter 112 that the content item is not classifiedby the threat data 114, then the processing node manager 118 may causeone or more of the data inspection engines 116 to perform the threatdetection processes to classify the content item according to a threatclassification. Once the content item is classified, the processing nodemanager 118 generates a threat data update that includes data indicatingthe threat classification for the content item from the threat detectionprocess, and transmits the threat data update to one of the authoritynodes 120.

The authority node manager 128, in turn, may update the master threatdata 124 and the master detection process filter 122 stored in theauthority node data store according to the threat data update receivedfrom the processing node 110. In an embodiment, the authority nodemanager 128 may automatically transmit the updated threat data anddetection processing filter to other processing nodes 110. Accordingly,threat data and the detection processing filter for new threats, as thenew threats are encountered, are automatically distributed to eachprocessing node 110, and each processing node 110 may update its localcopy of the detection processing filter 112 and threat data 114.

In a detection process filter and threat data pull and pushimplementation, each of the processing nodes 110 stores a detectionprocess filter 112, policy data 113, and threat data 114. The processingnode manager 118 accesses the detection process filter 112 to determinewhether the content item has been processed. If the processing nodemanager 118 determines that the content item has been processed, it maydetermine if the content item is classified by the threat data 114.Because the detection process filter 112 has the potential for a falsepositive, a lookup in the threat data 114 can be implemented to ensurethat a false positive has not occurred. The initial check of thedetection process filter 112, however, may eliminate many null queriesto the threat data 114, which, in turn, conserves system resources andincreases efficiency. If the processing node manager 118 determines thatthe content item has not been processed, it may request responsivethreat data for the content item from the authority node 120. Becauseprocessing a content item may consume valuable resource and time, insome implementations, the processing node 110 may first check with theauthority node 120 for threat data 114 before committing such processingresources.

The authority node manager 128 may receive the responsive threat datarequest from the processing node 110 and may determine if the responsivethreat data is stored in the authority node data 120 store. Ifresponsive threat data is stored in the master threat data 124, then theauthority node manager 128 provides a reply that includes the responsivethreat data to the processing node 110 so that the processing nodemanager 118 can manage the content item in accordance with the securitypolicy data 112 and the classification of the content item, and furtherupdate the local detection processing filter 112. Conversely, if theauthority node manager 128 determines that responsive threat data is notstored in the master threat data 124, then the authority node manager128 may provide a reply that does not include the responsive threat datato the processing node 110. In response, the processing node manager 118may cause one or more of the data inspection engines 116 to perform thethreat detection processes to classify the content item according to athreat classification. Once the content item is classified, theprocessing node manager 118 generates a threat data update that includesdata indicating the threat classification for the content item from thethreat detection process, and transmits the threat data update to anauthority node 120. The authority node manager 128 may then update themaster threat data 124. Thereafter, any future requests for related toresponsive threat data for the content item from other processing nodes110 can be readily served with responsive threat data.

The various push and pull data exchange processes provided above areexample processes for which the threat data and/or detection processfilters may be updated in the system 100 of FIGS. 1 and 2. Other updateprocesses, however, are contemplated with the present invention. Thedata inspection engines 116, processing node manager 118, authority nodemanager 128, user interface manager 132, logging node manager 148, andauthority agent 180 may be realized by instructions that upon executioncause one or more processing devices to carry out the processes andfunctions described above. Such instructions can, for example, includeinterpreted instructions, such as script instructions, e.g., JavaScriptor ECMAScript instructions, or executable code, or other instructionsstored in a non-transitory computer-readable medium. Other processingarchitectures can also be used, e.g., a combination of speciallydesigned hardware and software, for example.

§ 3.0 Example Server Architecture

FIG. 3 is a block diagram of a server 300 which may be used in thesystem 100, in other systems, or standalone. Any of the processing nodes110, the authority nodes 120, and the logging nodes 140 may be formedthrough one or more servers 300. Further, the computer device 220, themobile device 230, the servers 208, 216, etc. may include the server 300or a similar structure. The server 300 may be a digital computer that,in terms of hardware architecture, generally includes a processor 302,input/output (I/O) interfaces 304, a network interface 306, a data store308, and memory 310. It should be appreciated by those of ordinary skillin the art that FIG. 3 depicts the server 300 in an oversimplifiedmanner, and a practical embodiment may include additional components andsuitably configured processing logic to support known or conventionaloperating features that are not described in detail herein. Thecomponents (302, 304, 306, 308, and 310) are communicatively coupled viaa local interface 312. The local interface 312 may be, for example, butnot limited to, one or more buses or other wired or wirelessconnections, as is known in the art. The local interface 312 may haveadditional elements, which are omitted for simplicity, such ascontrollers, buffers (caches), drivers, repeaters, and receivers, amongmany others, to enable communications. Further, the local interface 312may include address, control, and/or data connections to enableappropriate communications among the aforementioned components.

The processor 302 is a hardware device for executing softwareinstructions. The processor 302 may be any custom made or commerciallyavailable processor, a central processing unit (CPU), an auxiliaryprocessor among several processors associated with the server 300, asemiconductor-based microprocessor (in the form of a microchip orchipset), or generally any device for executing software instructions.When the server 300 is in operation, the processor 302 is configured toexecute software stored within the memory 310, to communicate data toand from the memory 310, and to generally control operations of theserver 300 pursuant to the software instructions. The I/O interfaces 304may be used to receive user input from and/or for providing systemoutput to one or more devices or components. The user input may beprovided via, for example, a keyboard, touchpad, and/or a mouse. Systemoutput may be provided via a display device and a printer (not shown).I/O interfaces 304 may include, for example, a serial port, a parallelport, a small computer system interface (SCSI), a serial ATA (SATA), afibre channel, Infiniband, iSCSI, a PCI Express interface (PCI-x), aninfrared (IR) interface, a radio frequency (RF) interface, and/or auniversal serial bus (USB) interface.

The network interface 306 may be used to enable the server 300 tocommunicate over a network, such as the Internet, the WAN 101, theenterprise 200, and the like, etc. The network interface 306 mayinclude, for example, an Ethernet card or adapter (e.g., 10BaseT, FastEthernet, Gigabit Ethernet, 10 GbE) or a wireless local area network(WLAN) card or adapter (e.g., 802.11a/b/g/n). The network interface 306may include address, control, and/or data connections to enableappropriate communications on the network. A data store 308 may be usedto store data. The data store 308 may include any of volatile memoryelements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM,and the like)), nonvolatile memory elements (e.g., ROM, hard drive,tape, CDROM, and the like), and combinations thereof. Moreover, the datastore 308 may incorporate electronic, magnetic, optical, and/or othertypes of storage media. In one example, the data store 1208 may belocated internal to the server 300, such as, for example, an internalhard drive connected to the local interface 312 in the server 300.Additionally, in another embodiment, the data store 308 may be locatedexternal to the server 300 such as, for example, an external hard driveconnected to the I/O interfaces 304 (e.g., SCSI or USB connection). In afurther embodiment, the data store 308 may be connected to the server300 through a network, such as, for example, a network-attached fileserver.

The memory 310 may include any of volatile memory elements (e.g., randomaccess memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), nonvolatilememory elements (e.g., ROM, hard drive, tape, CDROM, etc.), andcombinations thereof. Moreover, the memory 310 may incorporateelectronic, magnetic, optical, and/or other types of storage media. Notethat the memory 310 may have a distributed architecture, where variouscomponents are situated remotely from one another, but can be accessedby the processor 302. The software in memory 310 may include one or moresoftware programs, each of which includes an ordered listing ofexecutable instructions for implementing logical functions. The softwarein the memory 310 includes a suitable operating system (O/S) 314 and oneor more programs 316. The operating system 314 essentially controls theexecution of other computer programs, such as the one or more programs316, and provides scheduling, input-output control, file and datamanagement, memory management, and communication control and relatedservices. The one or more programs 316 may be configured to implementthe various processes, algorithms, methods, techniques, etc. describedherein.

§ 4.0 Example Mobile Device Architecture

FIG. 4 is a block diagram of a user device 400, which may be used in thesystem 100 or the like. The user device 400 can be a digital devicethat, in terms of hardware architecture, generally includes a processor402, input/output (I/O) interfaces 404, a radio 406, a data store 408,and memory 410. It should be appreciated by those of ordinary skill inthe art that FIG. 4 depicts the mobile device 410 in an oversimplifiedmanner, and a practical embodiment may include additional components andsuitably configured processing logic to support known or conventionaloperating features that are not described in detail herein. Thecomponents (402, 404, 406, 408, and 402) are communicatively coupled viaa local interface 412. The local interface 412 can be, for example, butnot limited to, one or more buses or other wired or wirelessconnections, as is known in the art. The local interface 412 can haveadditional elements, which are omitted for simplicity, such ascontrollers, buffers (caches), drivers, repeaters, and receivers, amongmany others, to enable communications. Further, the local interface 412may include address, control, and/or data connections to enableappropriate communications among the aforementioned components.

The processor 402 is a hardware device for executing softwareinstructions. The processor 402 can be any custom made or commerciallyavailable processor, a central processing unit (CPU), an auxiliaryprocessor among several processors associated with the mobile device410, a semiconductor-based microprocessor (in the form of a microchip orchipset), or generally any device for executing software instructions.When the mobile device 410 is in operation, the processor 402 isconfigured to execute software stored within the memory 410, tocommunicate data to and from the memory 410, and to generally controloperations of the mobile device 410 pursuant to the softwareinstructions. In an embodiment, the processor 402 may include anoptimized mobile processor such as optimized for power consumption andmobile applications. The I/O interfaces 404 can be used to receive userinput from and/or for providing system output. User input can beprovided via, for example, a keypad, a touch screen, a scroll ball, ascroll bar, buttons, barcode scanner, and the like. System output can beprovided via a display device such as a liquid crystal display (LCD),touch screen, and the like. The I/O interfaces 404 can also include, forexample, a serial port, a parallel port, a small computer systeminterface (SCSI), an infrared (IR) interface, a radio frequency (RF)interface, a universal serial bus (USB) interface, and the like. The I/Ointerfaces 404 can include a graphical user interface (GUI) that enablesa user to interact with the mobile device 410. Additionally, the I/Ointerfaces 404 may further include an imaging device, i.e., camera,video camera, etc.

The radio 406 enables wireless communication to an external accessdevice or network. Any number of suitable wireless data communicationprotocols, techniques, or methodologies can be supported by the radio406, including, without limitation: RF; IrDA (infrared); Bluetooth;ZigBee (and other variants of the IEEE 802.15 protocol); IEEE 802.11(any variation); IEEE 802.16 (WiMAX or any other variation); DirectSequence Spread Spectrum; Frequency Hopping Spread Spectrum; Long TermEvolution (LTE); cellular/wireless/cordless telecommunication protocols(e.g., 3G/4G, etc.); wireless home network communication protocols;paging network protocols; magnetic induction; satellite datacommunication protocols; GPRS; proprietary wireless data communicationprotocols such as variants of Wireless USB; and any other protocols forwireless communication. The data store 408 may be used to store data.The data store 408 may include any of volatile memory elements (e.g.,random access memory (RAM, such as DRAM, SRAM, SDRAM, and the like)),nonvolatile memory elements (e.g., ROM, hard drive, tape, CDROM, and thelike), and combinations thereof. Moreover, the data store 408 mayincorporate electronic, magnetic, optical, and/or other types of storagemedia.

The memory 410 may include any of volatile memory elements (e.g., randomaccess memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), nonvolatilememory elements (e.g., ROM, hard drive, etc.), and combinations thereof.Moreover, the memory 410 may incorporate electronic, magnetic, optical,and/or other types of storage media. Note that the memory 410 may have adistributed architecture, where various components are situated remotelyfrom one another, but can be accessed by the processor 402. The softwarein memory 410 can include one or more software programs, each of whichincludes an ordered listing of executable instructions for implementinglogical functions. In the example of FIG. 4, the software in the memory410 includes a suitable operating system (O/S) 414 and programs 416. Theoperating system 414 essentially controls the execution of othercomputer programs and provides scheduling, input-output control, fileand data management, memory management, and communication control andrelated services. The programs 416 may include various applications,add-ons, etc. configured to provide end-user functionality with the userdevice 400. For example, example programs 416 may include, but notlimited to, a web browser, social networking applications, streamingmedia applications, games, mapping and location applications, electronicmail applications, financial applications, and the like. In a typicalexample, the end-user typically uses one or more of the programs 416along with a network such as the system 100.

§ 5.0 Example General Cloud System

FIG. 5 is a network diagram of a cloud system 500 for implementing thesystems and methods described herein for tracking and auditing changesin a multi-tenant cloud system. The cloud system 500 includes one ormore cloud nodes (CN) 502 and central authority (CA) nodes 506communicatively coupled to the Internet 504. The cloud nodes 502 mayinclude the processing nodes 110, the server 300, or the like. Thecentral authority nodes 506 may include the authority nodes 120, theserver 300, or the like. That is, the cloud system 500 may include thedistributed security system 100 or another implementation of acloud-based system, such as a system providing different functionalityfrom security. In the cloud system 500, traffic from various locations(and various devices located therein) such as a regional office 510,headquarters 520, various employee's homes 530, mobile laptop 540, andmobile device 542 communicates to the cloud through the cloud nodes 502.That is, each of the locations 510, 520, 530, 540, 542 iscommunicatively coupled to the Internet 504 through the cloud nodes 502.For security, the cloud system 500 may be configured to perform variousfunctions such as spam filtering, uniform resource locator (URL)filtering, antivirus protection, bandwidth control, data lossprevention, zero-day vulnerability protection, web 2.0 features, and thelike. In an embodiment, the cloud system 500 and the distributedsecurity system 100 may be viewed as Security-as-a-Service through thecloud. In general, the cloud system 500 can be configured to perform anyfunction in a multi-tenant environment. For example, the cloud system500 can provide content, a collaboration between users, storage,application hosting, and the like.

In an embodiment, the cloud system 500 can utilize the systems andmethods for tracking and auditing changes in a multi-tenant cloudsystem. That is, the cloud system 500 can track and audit administratoractivity associated with the cloud system 500 in a segregated andoverlaid fashion from the application functions performed by the cloudsystem 500. This segregated and overlaid fashion decouples the trackingand auditing from application logic, maximizing resources, andminimizing development complexity and runtime processing. The cloudsystem 500 (and the system 100) can be offloaded from complex trackingand auditing functions so that it can provide its primary function. Inthe context of a distributed security system, the tracking and auditingsystems and methods enable accountability, intrusion detection, problemdiagnosis, and data reconstruction, all in an optimized fashionconsidering the exponential growth in cloud-based systems.

There various techniques to forward traffic between users (locations510, 520, 530, devices 540, 542) and the cloud system 500. Typically,the locations 510, 520, 530 can use tunneling where all traffic isforward, and the devices 540, 5420 can use an application, proxy, SecureWeb Gateway (SWG), etc. Additionally, the cloud system 500 can bemulti-tenant in that it operates with multiple different customers(enterprises), each possibly including different policies and rules. Oneadvantage of the multi-tenancy and a large volume of users is thezero-day/zero-hour protection in that a new vulnerability can bedetected and then instantly remediated across the entire cloud system500. Another advantage of the cloud system 500 is the ability for thecentral authority nodes 506 to instantly enact any rule or policychanges across the cloud system 500. As well, new features in the cloudsystem 500 can also be rolled up simultaneously across the user base, asopposed to selective upgrades on every device at the locations 510, 520,530, and the devices 540, 542.

§ 6.0 DNS Augmented Security

In an embodiment, the cloud system 500 and/or the distributed securitysystem 100 can be used to perform DNS surrogation. Specifically, DNSsurrogation can be a framework for distributed or cloud-basedsecurity/monitoring, as is described herein. Endpoint security is nolonger effective as deployments move to the cloud with users accessingcontent from a plurality of devices in an anytime, anywhere connectedmanner. As such, cloud-based security is the most effective means toensure network protection where different devices are used to accessnetwork resources. Traffic inspection in the distributed security system100 and the cloud-based system 500 is performed in an in-line manner,i.e., the processing nodes 110 and the cloud nodes 500 are in the datapath of connecting users. Another approach can include a passiveapproach to the data path. DNS is one of the most fundamental IPprotocols. With DNS surrogation as a technique, it is proposed to useDNS for dynamic routing of traffic, per-user authentication and policyenforcement, and the like.

In conjunction with the cloud system 500 and/or the distributed securitysystem 100, various techniques can be used for monitoring which aredescribed on a sliding scale between always inline to never inline.First, in an always inline manner, all user traffic is between inlineproxies such as the processing nodes 110 or the cloud nodes 502 withoutexception. Here, DNS can be used as a forwarding mechanism to the inlineproxies. Second, in a somewhat always inline manner, all user trafficexcept for certain business partners or third parties is between inlineproxies such as the processing nodes 110 or the cloud nodes 502. Third,in an inline manner for most traffic, high bandwidth applications can beconfigured to bypass the inline proxies such as the processing nodes 110or the cloud nodes 502. Example high bandwidth applications can includecontent streaming such as video (e.g., Netflix, Hulu, YouTube, etc.) oraudio (e.g., Pandora, etc.). Fourth, in a mixed manner, inlinemonitoring can be used for “interesting” traffic as determined bysecurity policy with other traffic being direct. Fifth, in an almostnever inline manner, simple domain-level URL filtering can be used todetermine what is monitored inline. Finally, sixth, in a never inlinemanner, DNS augmented security can be used.

FIG. 6 is a network diagram of a network 550 with a distributed securitycloud 552 providing DNS augmented security. The network 550 includes auser device 554 connecting to the distributed security cloud 552 via ananycast DNS server 556. The anycast DNS server 556 can be a server suchas the server 300 of FIG. 3. Also, the anycast DNS server 556 can be theprocessing node 110, the cloud node 502, etc. The distributed securitycloud 552 includes the anycast DNS server 556, policy data 558, and aninline proxy 560. The inline proxy 560 can include the processing node110, the cloud node 502, etc. In operation, the user device 554 isconfigured with a DNS entry of the anycast DNS server 556, and theanycast DNS server 556 can perform DNS surrogation as is describedherein. The distributed security cloud 552 utilizes the anycast DNSserver 556, the policy data 558, and the inline proxy 560 to perform theDNS augmented security.

The network 550 illustrates the DNS augmented security where DNSinformation is used as follows. First, at step 562, the user device 554requests a DNS lookup of a site, e.g., “what is the IP address ofsite.com?” from the anycast DNS server 556. The anycast DNS server 556accesses the policy data 558 to determine the policy associated with thesite at step 564. The anycast DNS server 556 returns the IP address ofthe site based on the appropriate policy at step 566. The policy data558 determines if the site either goes direct (step 568) to theInternet, is inspected by the inline proxy (step 570), or is blocked perpolicy (step 572). Here, the anycast DNS server 556 returns the IPaddress with additional information if the site is inspected or blocked.For example, if the anycast DNS server 556 determines the access isdirect, the anycast DNS server 556 simply returns the IP address of thesite. If the anycast DNS server 556 determines the site is blocked orinspected, the anycast DNS server 556 returns the IP address to theinline proxy 560 with additional information. The inline proxy 560 canblock the site or provide fully inline proxied traffic to the site (step574) after performing monitoring for security.

The DNS augmented security advantageously is protocol andapplication-agnostic, providing visibility and control across virtuallyall Internet-bound traffic. For example, DNS-based protocols includeInternet Relay Chat (IRC), Session Initiation Protocol (SIP), HypertextTransfer Protocol (HTTP), HTTP Secure (HTTPS), Post Office Protocol v3(POP3), Internet Message Access Protocol (IMAP), etc. Further, emergingthreats are utilizing DNS today, especially Botnets and advancedpersistent threats (APTs). For example, Fast flux is a DNS techniqueused to hide phishing and malware delivery sites behind an ever-changingnetwork of compromised hosts acting as proxies. The DNS augmentedsecurity provides deployment flexibility when full inline monitoring isnot feasible. For example, this can be utilized in highly distributedwith high bandwidth environments, in locations with challenging InternetAccess, etc. The DNS augmented security can provide URL filtering,white/blacklist enforcement, etc. for enhanced security without contentfiltering. In this manner, the network 550 can be used with thedistributed security system 100 and the cloud system 500 to providecloud-based security without requiring full inline connectivity.

§ 7.0 Multi-Tenant, Cloud-Based Firewall

FIG. 7 is a network diagram of a network 600 with a firewall 602 inaccordance with the multi-tenant cloud-based firewall systems andmethods. The firewall 602 is functionally deployed through the cloudsystem 500 where traffic from various locations (and various deviceslocated therein) such as a regional office/Branch office 510,headquarters 520, various employee's homes 530, mobile laptop 540, andmobile device 542 communicates to the Internet 504 through the cloudnodes 502. The firewall 602 can be implemented through the cloud node502 to allow or block data between the users and the Internet 504. Thefirewall 602 could also be implemented through the processing node 110.Note, in the various descriptions that follow, reference is made to thecloud node 502, but those of ordinary skill in the art will recognizethe processing node 110 can be used as well or any other type of serveror node. Further, the firewall 602 can be communicatively coupled to alog 604 for logging associated data therein. In an embodiment, the cloudnodes 502 can be used only to provide the firewall 602. In anotherembodiment, the cloud nodes 502 can provide the firewall 602 as well asin-line inspection. The firewall 602 can handle various types of data,such as, for example, Session Initiation Protocol (SIP), InternetMessage Access Protocol (IMAP), Internet Relay Chat (IRC), Simple MailTransfer Protocol (SMTP), Secure Shell (SSH), and the like.

Users connect to the cloud system 500 via Internet Protocol Security(IPsec) or GRE, all traffic, including non-HTTP traffic may be sentthrough the cloud nodes 502. The firewall systems and methods propose toadd support for non-HTTP applications to the cloud nodes 502. Thus, thecloud system 500 is able to support non-Web traffic and act as aFirewall for the Branch office, where clients typically sit behind ahardware-based firewall to connect to servers outside the hardware-basedfirewall. The firewall 602 provides advanced security functionality inthe cloud that can be used to offload Branch office Customer PremisesEquipment (CPE).

Advantageously, in the cloud system 500, processor andresource-intensive features are scalable, efficiently used for multiplecustomers, and inexpensive, relative to on-premises hardware-basedsolutions. The firewall 602 can be used to replace traditional expensiveappliance box solutions that reside at the customer premise with servicefrom the cloud system 500. This enables end customers to realize costsavings, provide efficient growth, and unified management/reporting. Forexample, the cloud system 500 scales while appliance box solutions donot. On-premises hardware-based solutions are often integrated withfeature-rich routers or operate as a stand-alone device. In bothscenarios, the firewall 602 can provide cost savings, either removingthe need for the stand-alone device or allowing the use of lower-costrouters and/or lower cost firewalls. The firewall 602, through the cloudsystem 500, can offer granular Layer 3 (L3) through Layer 7 (L7) controlof applications, in a multi-tenant cloud infrastructure. This alsoincludes integrated logging functionality, giving customers visibilityinto applications down to the L3 applications running on their networks.

§ 7.1 Multi-Tenant, Cloud-Based Firewall—Use Cases

FIG. 8 is a network diagram of a network 600A illustrating example usecases of the firewall 602. Here, the firewall 602 can support multiplecustomers, such as regional office/Branch offices 510A, 510B. That is,the cloud system 500 can support the firewall 602 for more than onecustomer at a time. Additionally, the firewall 602 can support a roadwarrior, i.e., the user device 554, outside the office.

In an embodiment, the firewall 602 can be an outbound firewall for aBranch office, such as for large distributed enterprises, medium-sizebusiness, small business, and the like, with users sitting behind thefirewall 602 and connecting to the cloud nodes 502 which allow outboundconnections of various protocols. Traffic can come to the firewall 602via IPSEC or GRE tunnels. Traffic can also come to the firewall 602 in aLayer 2 (L2) Transparent Mode, where Virtual Local Area Network (VLAN)tags are used, such as a specific tag mapped to a particular customer.In another embodiment, the firewall 602 can be an outbound firewall forBranch offices for Managed Service Provided, replacing existing managedfirewall servers where large amounts of appliances are installed in datacenters.

The firewall 602 also can provide basic stateful firewall functionalityfor common Layer 3 (L3) applications, allowing for the configuration ofany one of these applications to traverse through the firewall 602. Theuser will now be capable of managing and controlling which protocols andapplications are allowed through the firewall 602 and which ones aredropped.

For example, Telnet traffic can be configured to be allowed, and allother non-HTTP/HTTPS traffic to be dropped. Inbound functionality or anyconnections initiated by users coming from the Internet 504 can also besupported. The firewall 602 also includes an ability to supportconfiguration policy rules and the ability to log all traffic andgenerate reports for the customer.

In an embodiment, the regional office/Branch offices 510A, 510B can eachconnect to the cloud system via an IPSec tunnel, configured for alltraffic, including non-HTTP/HTTPS. This traffic can be Network AddressTranslation (NAT) out to the Internet 504, and return traffic is passedback through the appropriate tunnel. Because the cloud system 500 knowswhich customer and which location traffic originated, the return trafficcan be mapped properly and sent back through the appropriate VPN tunnel,even though customers may have overlapping private address spaces.

§ 7.2 Multi-Tenant, Cloud-Based Firewall—Functionality

A firewall service is defined to be a traditional Layer 4 (L4) servicethat can be defined by ports and Ethernet protocol (Telnet, SSH, POP,IMAP, etc.). Firewall applications are defined as Layer 7 (L7)applications (e.g., Lync, Skype, YouTube, etc.). The firewall 602enables custom firewall services to allow users to define their ownpinholes through the FW firewall 602 if a pre-defined firewallapplication does not exist. This will be known as a custom-definedapplication that requires support for the custom application name andthe configuration of ports or port ranges. This custom-definedapplication can override any pre-defined applications, and thecustom-defined application cannot be defined with conflicting portranges.

The firewall 602 can support pre-defined applications including, but notlimited to, the following:

HTTP Port 80 HTTPS Port 443 SMTP Port 25 File Transfer Protocol (FTP)Port 21 control, Port 20 data control and Data ICMP Telnet Port 23 DNSPort 53 Network Time Protocol (NTP) Port 123 (User Datagram Protocol(UDP)) SSH Port 22 Post Office Protocol (POP) Ports 109/110 IMAP Ports143/220 Remote Procedure Call Port 111 SNMP Ports 161 (UDP)/162(TCP/UDP) BGP ActiveSync Secure SMTP (SSMTP) Port 465 Secure IMAP(IMAP4-SSL) Port 585 IMAP4 over SSL (IMAPS) Port 993 Secure POP3(SSL-POP) Port 995

The firewall 602 can also support HTTP/HTTPS on non-standard portsthrough customer definition.

§ 7.3 Application Support

The firewall 602 can provide application signature support, whichprovides the visibility necessary for administrators to understand theapplications running on the network including firewall services andapplications. The application signature can detect a set of applicationsvia a compiled signature database. The signatures are grouped intodefault groups with individual apps added to the appropriate group. Auser can define a custom group and define which group an applicationresides. Signatures for custom applications are user-definable(typically through a Regular Expression (regex) engine).

FIG. 9 is a screenshot associated with the firewall 602 illustratingexample network services. Specifically, the firewall 602 includesseveral predefined services based on ports. Further, users can createtheir own customer services and service groups. FIG. 10 is a screenshotassociated with the firewall 602 illustrates example applications. In anembodiment, the firewall 602 can support thousands of applications(e.g., approximately 1200 applications), covering Peer-to-Peer (P2),Instant Messaging (IM), port evasive applications, streaming media, andother applications. Again, because the firewall 602 is multi-tenant anddistributed (e.g., worldwide), new services and applications can beadded instantly, across all customers and locations.

FIG. 11 is a diagram of a Deep Packet Inspection (DPI) engine 650 forthe firewall 602. The DPI engine 650 is part of or works with thefirewall 602 to categorize incoming packets 652 to the firewall 602. TheDPI engine 650 includes application plugins 654, application ID metadata656, and flow processing 658. The application plugins 654 is configuredto receive application updates 660 that can be regularly or periodicallyprovided by the cloud system 500. Through the application updates 660,the firewall 602 can support more than the approximately 1200applications. The application ID metadata 656 provides details on howdifferent applications are detected. The flow processing 658 operates onthe incoming packets 652 using the application ID metadata 656 todetermine applications associated with the incoming packets 652. Theflow processing 658 identifies the protocol and application behind eachIP flow of the packets 652 using stateful inspection and heuristicanalysis through the extraction of metadata from protocols (e.g., appinfo, volume, jitter) and does not require SSL decryption. If the DPIengine 650 cannot classify app traffic, it will be categorized as eitherTCP, UDP, HTTP, or HTTPS. The DPI engine 650 can provide reporting 672data to the log 604 as well as receive policy 674 updates.

The DPI engine 650 can use various classification methods, includingexplicit, Protocol Data Signature(s), Port-based classification overSSL, IP protocol number, pattern matching, session correlation, and thelike. Explicit classification is at a bottom layer where a protocol isidentified by information found in the layer below. For example, the IPprotocol includes a field called “protocol” defining the protocolembedded in its payload. The Protocol Data Signature(s) is through aProtocol Data Engine. When parsing the HTTP, SSL, and Real-TimeMessaging Protocols (RTMP) protocol headers, the Protocol Data Enginecan look at a combination of specific value such as HTTP:Server,HTTP:Uniform Resource Indicator (URI), HTTP:User_agent,RTMP:page_Uniform Resource Locator (URL), SSL:common_name, andclassifies the upper protocol using this information. For example,Facebook is classified after seeing an HTTP host matching *.facebook.comor *.fbcdn.net. In an embodiment, the DPI engine 650 was shown to takeabout 20 packets in order to detect the application.

For Port-based classification over SSL, in order to classify flows ontop of SSL, the TCP port can be used in order to differentiate HTTPS,IMAPS, POP3, etc. For example, POP3 is classified in the SSL TCP port995. For IP protocol number, this is a subset to the explicitclassification for protocols above IP. As described above, protocolsabove IP are explicitly specified in the IP protocol. For patternmatching, content parsing is used to identify the protocol. For example,the pattern matching searches for multiple patterns such asHTTP/1.[0|1], [GET | POST | HEAD | CONNECT | PUT | DELETE], and thelike. For session correlation, information is required extracted fromanother flow in which the other protocol negotiated an IP and port foropening a new flow. For example, FTP-data by itself is only a binarystreamed over the network and does not provide any information forclassification. The only way to classify it is by using information fromthe FTP session leading to the opening of this flow in which FTP isspecifying the IP and port to use for the ftp_data session.

FIG. 13 is screenshots of editing IP groups. Specifically, FIG. 13illustrates editing a source IP group and editing a destination IPgroup. IP groups can be predefined for the internal network anddestination IPs. Destination IPs can be configured with IP-basedcountries and IP categories. FIG. 14 is screenshots of editing a networkservice. The firewall 602 can include editing HTTP and HTTPS networkservices to include non-port 80/443 ports, including configured portsthat are not used in other services.

§ 8.0 Policy

A firewall policy (or rule) is an exact description of what the firewall602 is supposed to do with particular traffic. When enabled, thefirewall 602 always have at least one active rule, although usuallymultiple rules are employed to differentiate traffic varieties by{source, destination, and application} and treat them differently. Ingeneral, firewall policy consists of matching criteria, an action, andsome attributes:

rule_rank rule label [who] [from] [to] [network service] [networkapplication] [when] action [action restrictions] [rule status] [logging]

The firewall 602 supports a policy construct, to determine wherefirewall policy is enforced during an overall order of operation ofpacket flow through the cloud node 502. In an embodiment, there arethree types of policy, namely, firewall policy, NAT policy, and DNSpolicy.

The firewall policy construct supports a rule order, status, criteria,and action. Policies are matched in the rule order in which they weredefined. The status is enabled or disabled. The matching criteria caninclude the following:

From Location, Department, Group, IP Address, IP Address Group, IPaddress Ranges, User, and/or User Group To IP address, Address Group,Domain Name or countries Firewall service(s) L4 services as listedabove, and new services may be defined by Source IP, Destination IP,Source Port, Destination Port, and Protocol Firewall application(s) L7application supported by a Deep Packet Inspection (DPI) engine WhenSchedule Daily quota Time or bandwidth, allowing the user to configurethe amount of time or bandwidth a user is allowed for a certainapplication. Action Allow or block by either dropping traffic or bysending TCP reset

All components of the matching criteria are optional and if skippedimply “any.” A session matches a rule when all matching criteriacomponents of the rule are satisfied (TRUE) by the session. If a sessionmatches any element of a component (i.e., one of the IPs in a group),then the entire component is matched.

A rule might be configured as either company-wide or restricted to up toa certain number of locations, or up to a certain number of departments,or up to a certain number of users. Some rules might extend theircoverage to the entire cloud (SNATor tracking rules), applying to everycompany in the cloud. Source/destination IPs are a group of thefollowing in any number/combination. It is used to match sessionsource/destination IPs: •individual IP, i.e. 192.168.1.1; •IP sub-net,i.e. 192.168.1.0/24; •IP range, i.e. 192.168.1.1-192.168.1.5. Note thatthere is no special support for IP range exclusions; •IP category. Sameas the URL category and comes from a database. Custom categories aresupported. Applicable to destination IPs only; •country—matches any IPthat belongs to this country, i.e., “Russia.” Applicable only todestination IPs; •domain name—any destination IPs behind this namematches this criterion. For example, any IP that matches “skype.com.”The data plane builds an IP cache to match the names from DNS requestscoming from the clients.

A network service is a group of {TCP/UDP, {src/dst port(s), or portranges, or port sets}} or just ICMP. Network service defines anapplication based on L3/L4 information of the first packet in a session.Following is restrictions and implementation details: Each networkservice can be identified either by its name (aka label) or invisiblefor customer slot number in the range from 0 to 127. The slot number isrequired for firewall logging. Following slot numbers are reserved tosimplify data plane implementation: •0—predefined customizable HTTPservice group; •1—predefined customizable HTTPS service group;•2—predefined customizable DNS service group; •3-5—reserved for futureuse; •6—ICMP any. This service covers ANY ICMP traffic; •7—UDP any. Thisservice covers ANY UDP traffic—port from 0 to 65535; •8—TCP any. Thisservice covers ANY TCP traffic—port from 0 to 65535; •63—OTHER. Thisservice covers all network services that don't match any predefined orcustom services. Basically, it will catch all protocols other than ICMP,UDP, and TCP. •64—is the very first slot of the custom services. Thecustomer can be allowed to alter (add, modify or delete) protocol andports in all predefined services except ICMP any, UDP any, TCP any, andOTHER. Although the customer is not allowed to delete a predefinednetwork service, or modify its name, or delete all protocol/port entriesin a particular predefined service. Different services must not haveoverlapping ports for the same protocols. The only exception ispredefine •any services. The data plane chooses more specific networkservice for logging. For example, if the session matches 2 networkservices TCP any and SSH then SSH is logged for this session.

The network application is defined based on L7 info. This ispreconfigured for the cloud and comes only from the DPI engine 650. Rankis the priority of the rule. Rank is needed to resolve conflicts when asession matches more than one rule. The highest priority rule (the leastrank number) takes precedence;

A Rule's action defines what should be done with the matching session.There might be several actions required to apply to a single session.For example, the first action lets the session go through (allow), anext action tells of tracking the session using state-full TCP proxy,next is to apply source NAT to the session, and final action redirectsthe session to a preconfigured IP. All these different actions belong todifferent rules. In other words, if the firewall 602 can apply up to 4different actions to a single session, it's required to fetch up to 4different policies for that session. To minimize the number of rulesshown to the user front end might want to plump different rule typesinto a single rule as far as matching criteria are the same for thoserules.

Here are example supported types of policies categorized by action type:•filtering policies—to allow or block sessions; •tracking policies—tellhow to track allowed sessions—state-fully or statelessly; •SNATpolicies—dictate how to apply source NAT; •DNAT policies—configuresdestination NAT; •bandwidth control policies; •DNS policies—provideDNS-specific actions. Tracking, SNAT and DNAT policies must be enforcedat the first packet. Hence, they do not support network applicationmatching component since its evaluation takes several packets. Actionrestrictions allow to modify rule action depending on some dynamic info.For example, the customer might want to limit total time or bytes perday of youtube.com traffic.

Depending on the action, there are different types of rules. Thefollowing types can be supported: •filtering rules are evaluated first.Monitored rule status overrides (only) filtering rules action—makes itallow without any restrictions. This type of rule is user-configurable.They provide the following actions: •allow—pass to the evaluation ofother types of rules. This action might have an additional restrictionfor daily time/bandwidth quota; •block_drop—silently drop all packetthat matches the rule; •block_reset—for TCP sessions send TCP reset tothe client. For non-TCP traffic act the same as block_drop; andblock_icmp—response to the client with ICMP error message type 3(Destination unreachable), code 9 or 10 (network/host administrativelyprohibited).

Tracking rules provide—state-full or stateless action. Only OPsconfigure this type of rules; they are hidden from the user. Thegranularity of who component in the matching criteria should be fromuser to cloud wide.

SNAT rules dictate which type of outbound IP should be used for all thetraffic matching such rule. Two types of outbound IPs are supported—openand secure. SNAT rules are applied to all outbound traffic, and there isno way to disable it. These rules might be configured by OPs only. Theonly purpose of SNATrules is to isolate harmful traffic from the rest ofthe clients. Requires persistence on the node 110. The granularity ofwho component in the matching criteria should be from user to cloudwide.

DNAT (redirect) rules provide destination IP and port (as the actionattribute). They tell where the client-side traffic has to beredirected. Port is optional and when is not specified firewall does notalter destination port. DNAT is user-configurable.

A Rule's attributes include: •rule rank—reflects the priority of therule comparing to the other rules; •rule label—rule specific label (orname) which is shown in firewall reports. This is a way to matchconfiguration and reporting; •rile status—administrative status of therule—enabled, disabled, or monitor; •logging—tells how to log sessionscreated via this rule.

The NAT policy construct includes source NAT and destination NAT. Forthe source NAT, all applications, including custom defined applications,are NAT'ed with a public IP address associated with the cloud system 500(source NAT'ed). All return traffic is received and sent back to theappropriate IPsec or GRE tunnel. It may be desirable from an operationsperspective to have a different IP address for firewall source NAT'dtraffic that for HTTP(S) source NAT'd traffic. This is to avoidblacklists between the two functionalities, so the firewall 602customers do not accidentally blacklist our HTTP only customers. Fordestination NAT (DNAT), in cases where the customer wants to force aprotocol out a particular port DNAT will be required.

The DNS policy construct includes the following:

To IPs and countries IP/domain category Group of IP or domain categoriesderived Network service Network application Action Allow, block,redirect_request (to a different DNS server or substitute IP in responsewith pre-configured IP)

DNS might be policed on the session as well as on transaction(individual request) levels. While session DNS policies have regularpolicy structure, the DNS transaction policies are different:

rule_label [who] [from] [to] [IP/domain category] [when] action [actionrestrictions] [rule status][logging]

The differences are: •to—a group of IPs and countries. Note that IPcategories should not be included here to avoid confusion. These are IPsor countries of the destination DNS server; •IP/domain category—a groupof IP or domain categories derived from ZURL DB. These categories arederived from matching the DNS request domain or responded IP a database.Such separation of to (server IP) and IP/domain category allows toconfigure fine granular matching criteria like “malicious IP/domainrequest sent to specific DNS server”; •network service—is notconfigurable here because DNS transaction policies get applied only tothe sessions that matched predefined DNS service group; •networkapplication—is not configurable. There is no way to find an applicationjust by IP (w/o port/protocol). This finds out the application when aclient comes with a session using a resolved IP as destination IP.Besides URL category lookup does not return application ID. Theapplication requires one extra look up; •action—actions applied only toDNS transactions. It includes allow, block, redirect_request (redirectto a different DNS server), redirect_response (substitute IP in responsewith pre-configured IP). Rules with redirect_request action can beapplied only to the request phase of DNS transaction. Rules withredirect_response action are applicable only to the response phase ofDNS transaction. And finally, rules with allow or block actions areevaluated during both phases (request and response) of DNS transaction.

The firewall 602 can support various policies, e.g., 128 policies, 1024policies, etc., including variable locations, departments, and users perpolicy. Again, since the firewall 602 is multi-tenant, policies can bedifferent for each customer as well as different for differentlocations, departments, and users per customer. For user-based policy, aspecific user must have IP surrogation enabled for user tracking. FIG.12A is a screenshot of defining a firewall filtering rule. The rule isnamed, has an order and rank, and is enabled/disabled. Matching criteriaare set for the users, groups, departments, locations—Who, From, To,Network Service, Network App, When. Finally, the action isdetermined—Allow, Block/Drop, Block with ICMP Error Response, Block withTCP Reset. FIG. 12B is another screenshot of defining a firewallfiltering rule. Network service and network application criteria in thesame rule results in a logical “AND” condition. In FIG. 12B, a Telnetnetwork service on Port 23 and a Telnet network application on anyport—“AND” results in telnet protocol as detected by the DPI engine 650must be on port 23. Conversely, criteria within the same network serviceor network app are logical “OR.”

FIG. 15 is a flow diagram of packet flow through the cloud node 502.Again, all traffic 680 between users and the Internet 504 is processedthrough the cloud node 502 (or the processing node). The traffic 680 canbe received at a Location Based (LB) instance 682 which could alsoreceive traffic from GRE, a Virtual IP (VIP) IPsec, an LB VIP, etc. Fromthe LB instance 682, the traffic 680 is sent to one or more instances684, 686, 688 (labeled as instance #1, #2, #3). For illustrationpurposes, the instance 686 is shown which includes a firewall engine690, a Web engine 692, and a policy engine 694. The firewall engine 690forwards on port 80/443 traffic to the policy engine 694 and port 80/443traffic to the Web engine 692. If Web policy and FW policy areconfigured for a Web application, Web policy is applied first and thenFW policy will be enforced. The policy engine 694 is configured toenforce Web and firewall policies and to send the traffic 680 to theInternet 504.

FIG. 16 is a flowchart of a process 700 for packet flow through thefirewall 602 from a client. The process 700 includes receiving a packet(step 702). If the packet is from a tunnel (step 704), the packet isde-encapsulated (step 706) and the process 700 returns to step 702.After step 704, a firewall session lookup is performed (step 708). If nofirewall session exists, the process 700 checks if the packet islocation-based (step 710). If the packet is not location-based (step710) and not port 80/443 traffic (step 712), the traffic is dropped(step 714). If the packet is location-based (step 710), the process 700checks if the packet is destined for the cloud node 502 (step 716) andif so, moves to step 712. If the packet is not destined for the cloudnode 502 (step 716), the process 700 includes creating a firewallsession (step 718). After step 718 and if a firewall session exists instep 708, the process 700 checks if the traffic is port 80/443 (step720), and if so, established a web proxy (step 722). After steps 716,722, the process 700 checks if the location firewall is enabled (step724). If so, the traffic is processed by the firewall engine 690, and ifnot, the traffic is NAT'd (step 726). The firewall engine 690 analyzesthe traffic through a network services/DPI engine (step 728), appliesfirewall policy (step 730), and the traffic is NAT'd (step 726).Finally, the packet is sent (step 732).

FIG. 17 is a flowchart of a process 750 for packet flow through thefirewall 602 from a server. The process 750 includes receiving a packet(step 752), and a firewall session lookup is performed (step 754). If nosession exists, the process checks if the traffic is port 80/443 (step756), and if not, drops the traffic (step 758). If the traffic is port80/443 (step 756), a firewall session is created (step 760) and a Webproxy is performed (step 762). The process 750 checks is locationfirewall is enabled (step 764). If so, the traffic is processed by thefirewall engine 690, and if not, the traffic is NAT'd (step 766). Thefirewall engine 690 analyzes the traffic through a network services/DPIengine (step 768), applies firewall policy (step 770), and the trafficis NAT'd (step 766). Finally, the packet is sent (step 772).

Every packet hits the firewall 602 which requires the firewall 602 toprocess packets as efficiently as possible. This is achieved by havingslow and fast paths for packet processing. The slow path deals with thevery first packet of a new session. It is slow because the correspondingpolicy has to be found and firewall resources allocated (memory, ports,etc.) for the session. All packets of an existing session go through afast path where only a simple lookup is required to find thecorresponding session.

Here is a description of policy evaluation of the first packet in asession—the slow path:

-   -   every packet hits the firewall code first—it is intercepted on        the ip_input( ) level;    -   if the packet destined to one of the cloud system 500's IP        addresses, a pass up session is created, and the packet is        forwarded up to the network stack. No firewall policy is        evaluated in this case;    -   the who component of matching criteria is evaluated based on a        combination of:        -   client IP— inner IP in case of a tunnel or just client IP in            case of L2 redirection; ∘tunnel info—outer IP of the tunnel;        -   default location IP if auth_default_location_ip is            configured to 0 value in sc.conf. This IP is used as            location IP and overrides any tunnel info;    -   based on the who value the following actions might be taken:        -   if the packet came for a road warrior (no location is found            for the client's IP) and status was ready at least once,            then pass up this session.        -   the packet came from a known location. If firewall            functionality is disabled for the company—a pass up session            is created, and the packet gets forwarded to the networks            stack;        -   if the firewall is not configured for the location, a new            session object gets created with allow the action, and the            packet are SNATed out. The session is allowed to overcome            rule infrastructure limitation of only up to 8 locations per            rule—the i.e. company wants to disable FW in 100 locations            out of 10000. Otherwise, firewall policy evaluation            continues;        -   if the firewall fails to retrieve company, location or user            configuration due to lack of resources (out of memory), then            the packet is silently dropped. If config retrieval failure            is due to any other reason then the cloud wide default            policy is applied to the session;        -   finally, if the firewall is configured for the client, and            all configuration is available the session is treated per            configured policies;    -   for policy lookup, firewall queries the configuration of the        corresponding company, location, location user, and if available        surrogate IP user. Company config contains a list of all        firewall rules. The location has the firewall enable/disable        knob. And the two users configs tell which firewall rules are        enabled for the particular location and particular surrogate IP        user.    -   policy lookup is done to find the highest priority best-matching        rule using all enabled rules for the location OR surrogate IP        user. In other words, policy lookup evaluated all rules enabled        for the location as well as for the surrogate IP user. Note that        if a user belongs to a company A while coming to the node 110        from the location of company B, then only location configured        policies are applied to such user;    -   to determine network application (which mostly comes from        layer 7) DPI engine usually has to see more than one packet.        That is why all filtering rules with “other-than-any” network        application components are replaced with similar rules where        network application is any and action is allowed. Based on the        result of the policy look up firewall creates a session object        and acts accordingly. For example, a rule from_subnet_1        network_application_tor DENY for the first look up gets replaced        with from_subnet_1 network_application_any ALLOW;    -   when several packets later network application are determined by        DPI, it notifies firewall about the findings. At this point, the        firewall checks the original (non-modified) policies and if        needed, can correct actions applied to the session. Using the        previous example, the DENY rule will be checked during this        second policy look up.

Again, all traffic is inspected through the cloud node 502. Web Traffic(Port 80/443) is sent to the Web Policy engine 692. If the firewall(non-port 80/443) is enabled, then all web traffic is sent to thefirewall engine 690 for inspection. Firewall traffic is sent to thefirewall engine 690 and will go through the firewall policy table. Webpolicies are inspected first. Firewall policies are enforced after allweb policies. If there is a web allow policy, firewall policies arestill evaluated.

FIG. 18 is a screenshot of creating firewall policies. The policies havean order, rule name, criteria, and action. Firewall policies start bydefining the Network Services to Allow followed by Network Applications.A basic policy is defined to allow HTTP, HTTPS, and DNS traffic justbefore the default rule. Again, it may take up to 20 packets in orderfor the DPI engine 650 to detect the Application. If a packet hits aNetwork Application policy, and the DPI engine 650 cannot determine theApplication, then the packet is allowed, and the next rule is notevaluated. The next rule will be evaluated once the Application isdetermined.

FIG. 19 is a screenshot of a NAT configuration. The firewall 602 cansupport destination NAT to redirect traffic to another IP and/or port.The use case is to control what resources a user can access. Forexample, a customer requires their users to go to an internal IP toaccess external non-web servers.

FIG. 20 is a screenshot of a user authentication screen. The userauthentication can leverage existing authentication infrastructure inthe systems 100, 500. In an embodiment, IP Surrogate is configured tomap the IP address to the user. The user must authenticate with the Webfirst (or have a cookie stored).

FIG. 21 is a screenshot of DNS policy. For example, the use case caninclude guest wireless where a sub location is created for a guestwireless to apply DNS-based policies. DNS policy includes an ability toapply policy based on DNS request—allow, block or redirect the request,redirect response. The DNS policy can be based on server IP orrequested/resolved IP category.

§ 9.0 Reporting and Logging

The firewall 602 can support the log 604. In an embodiment, the log 604can be through the logging nodes 140. The log 604 can be configurable.For example, by default, only blocked events or DNS events are logged.Aggregated logs can be used when logging exceeds certain thresholds orwhen large amounts of logs need to be processed and is of similartraffic type. For example, Internet Control Message Protocol (ICMP) logswill only be logged until a certain threshold, e.g., 10/second, and thenno additional logs will be sent until the traffic falls back under thethreshold. The definition of the thresholds for firewall sessions can bedefined.

Each firewall rule can be configured for full or aggregated logging.Full logging can be enabled by default on block policies. Aggregatelogging can be the default on for Allow rules. Allow rules can have theoption to be changed to Full logging. In another embodiment, two typesof log formats are enabled per rule—i) Full Session logging—performedfor all block firewall policies+DNS transactions, and ii) Hourly (orAggregate) logging—performed for Web logs to avoid duplication with Webtransactions.

A log format for the log 604 for firewall logs can include:

  Firewall instance ID Session Duration Time Stamp User DepartmentLocation Incoming Source IP Incoming Destination IP Incoming Source PortIncoming Destination Port Outgoing Source IP Outgoing Destination IPOutgoing Source Port Outgoing Destination Port Matched firewall rulesFirewall service Firewall application Action (Allow, block) Client TXBytes (from client to firewall 602)-Outbound Client RX Bytes (fromfirewall 602 to the client)-Inbound GRE or VPN IP Category Cloud node502 ID

A log format for the log 604 for DNS Request/Response logs can include:

Log Number Time User Department Location Source IP Destination IP  Query Domain IPS Category

A log format for the log 604 for Attack logs can include:

  Port Scan Syn Flood Tear Drop ICMP Flood UDP Flood WinNuke Etc.

The purpose of the reports is primarily two-fold, namely i) to providevisibility into the top Applications and Services that are traversingthe network and ii) to provide visibility into top firewall threats thathave been detected. Note, because the firewall 602 is multi-tenant anddistributed (e.g., worldwide), the visibility can be used to detectzero-day/zero-hour threats and instantly provide a defense.

Several reports can be supported to display the various fields above incolumns that can be configured to be visible or hidden in the display.The reports can be available based on a number of sessions or bytes. Theadmin can have the ability to filter based on the various fields. Thefilter can allow the admin (or other users) to show all sessions for adefined between for a particular user, IP address (Source orDestination), or group of IP addresses. There can be two types ofreports: Real-time reports generated by Compressed Stats and Analyzereports generated by full session log analysis. Each report below ismarked as (RT) Real-Time or (Analyze). Example reports can includeFirewall Usage Trend, Top firewall Applications (based # of sessions andbytes)—Includes Applications detected over HTTP, HTTPS (RT), Top BlockedRules Hit (RT), Top Internal Source IPs (Analyze), Top Destination IPs(Analyze), Top Users (RT), Top Departments (RT), Top Locations (RT),List of Top Users/Departments/Locations with Top Protocols for eachUser/Dept/Location, List of Top IPs with Top protocols per IP, Topfirewall Attacks (Analyze), etc.

FIG. 22 is a screenshot of a reporting screen for firewall insights.FIG. 23 is a screenshot of an interactive report for firewall insights.FIG. 24 is a screenshot of a graph of usage trends through the firewall602. FIG. 25 is graphs of top firewall protocols in sessions and bytes.

In an embodiment, a multi-tenant cloud-based firewall session loggingmethod performed by a cloud node includes firewall Session Stats loggingwhere are a firewall module records aggregated statistics based onvarious criterion such as client IP, user, network application,location, rule ID, network service, etc., firewall Session Full loggingwhere the firewall module records complete criterion such as userlocation, network application, customer location, rule ID, networkservice, client IP, etc. based on every session, and the firewall moduleimplements Rule based choice of logging as described herein.

In another embodiment, a multi-tenant cloud-based firewall withintegrated web proxy method is performed by a node in the cloud.Firewall traffic which is determined to be web traffic (default port80/443) is sent through the web proxy prior to being processed by thefirewall engine or non-web traffic (default non-port 80/443) traffic.Non-web traffic is processed by the firewall engine bypassing the webproxy engine. Rule order precedence of web traffic processed through theweb proxy policies before being processed by firewall policies. Thefirewall module integrated web proxy could reply End User Notificationpages if user traffic hits policies with action block.

§ 10.0 Cloud Firewall Deployment Modes

FIGS. 26 and 27 are network diagrams illustrating deployment modes ofthe cloud firewall 602. Specifically, FIG. 26 includes a web-onlybreakout for the cloud firewall 602, namely ports 80/443, and FIG. 27includes a full branch breakout where all traffic (all ports/protocols)is through the cloud firewall 602. Note, FIGS. 26 and 27 illustrate abranch office; of course, this can be a single user. The benefit of thecloud firewall 602 is it removes the requirement for local appliancesand provides customer IT control of the firewall 602 for the branchoffice.

As described herein, the cloud firewall 602 includes 1) Applicationawareness—Identify applications regardless of port, protocol, evasivetactic, or SSL using DPI engine; 2) User awareness—Identify users,groups, and locations, regardless of IP address; 3) Real-time, granularcontrol and visibility—Globally unified administration, policymanagement, and reporting; 4) Fully qualified domain name (FQDN)policies—Manage access policies for apps hosted on dynamic IPs(Azure/AWS) or across multiple IPs; and 5) Stateful firewallpolicies—Apply allow/block security policy based on source anddestination IP address, ports, and protocols.

Further, the cloud firewall 602 can be integrated with a Cloud Sandbox,web security, DLP, content filtering, SSL inspection, and malwareprotection, with cloud-scale correlation, reporting, and analytics, inthe cloud system 500 and/or the distributed security system 100.

The cloud firewall 602 includes a proxy-based architecture thatdynamically inspects traffic for all users, apps, devices, andlocations, natively inspects SSL/TLS traffic—at scale—to detect malwarehidden in encrypted traffic, and enables granular firewall policiesbased upon network app, cloud app, domain name (FQDN), and URL. Thecloud firewall 602 includes visibility and simplified management for IT,namely real-time visibility, control, and immediate policy enforcementacross the platform, logging of every session in detail, and the use ofadvanced analytics to correlate events and provide insights. The cloudfirewall 602 includes DNS security and control to protect users fromreaching malicious domains as the first line of defense, optimizes DNSresolution to deliver better user experience and cloud appperformance—critical for Content Delivery Network (CDN)-based apps, andprovides granular controls to detect and prevent DNS tunneling. Further,the cloud firewall 602 can support cloud-based IPS to deliver always-onIPS threat protection and coverage, regardless of connection type orlocation, to inspect all user traffic on and off network, even SSL, andSNORT style signature support.

§ 11.0 Cloud Node Architecture

FIG. 28 is a block diagram of functionality in the processing node 110or the cloud node 502 for implementing various functions describedherein. As described herein, the cloud firewall 602 is implemented bythe cloud system 500 and/or the distributed security system 100, via thecloud node 502 or the processing node 110. The cloud firewall 602provides a proxy-based firewall architecture, and FIG. 28 illustratesfunctional modules for supporting such architecture. A firewall modulecan provide Layer 3 (L3)-Layer 4 (L4) inspection, via a DPI engine, aDNS engine, an IPS engine, and a policy engine. A proxy module connectedto the firewall module can support Layer 7 (L7) capabilities, such as,without limitation, a sandbox engine, a DLP engine, bandwidth control,an AV/AS engine, a web IPS engine, ATP (Advanced Threat Protection), URLfiltering, and the like.

FIG. 29 is a block diagram and flowchart of how a packet is analyzedinside one of the processing nodes 110 or the cloud nodes 502. In thisexample, a user (e.g., at the regional office 510, headquarters 520,various employee's homes 530, the mobile laptop 540, and the mobiledevice 542, etc.) is accessing a cloud-based SaaS, such as Dropbox.Further, the description provided for FIG. 29 makes reference to thecloud system 500 and the cloud node 502, but is equally applicable tothe distributed security system 100 and the processing nodes 110.

The user, behind a gateway, sends traffic via a primary IPSEC tunnel tothe cloud system 500 for accessing the SaaS or the Internet 504 (stepS1). The cloud node 502 terminates the IPSEC tunnel and sends traffic toa load balancer and then to a cloud node 502 instance (step S2). Thecloud node 502 instance detects a specific app associated with thetraffic using DPI and sends to a proxy module (web) (step S3). The proxymodule inspects with URL filtering and DLP policies after SSL decryption(step S4). The traffic is NAT'd and sent to a server (step S5), and theuser IP is not exposed. Content is inspected on response and evaluatedfor APT, AV, and sandbox policies (step S6), firewall policy is enforced(step S7), and the traffic is returned encapsulated in the IPSEC tunnel(step S8) for the user to receive the traffic (step S9). Note, an IPSECtunnel may be used between the regional office 510, headquarters 520,etc. and the cloud system 500. Alternatively, an application can resideon the user device to forward traffic to the cloud system 500, insteadof the IPSEC tunnel.

Now, when traffic is sent to the cloud node 502 (or the processing node110), all traffic first hits the firewall engine. If the traffic is onport 80 or 443 or the firewall engine determines that this traffic (noton port 80 or 443) is web, then it will forward the traffic to the webengine for processing. Once the web engine has processed the traffic itis once again forwarded back to the firewall 602 and the firewall rulesare evaluated. If the traffic is not port 80 or 443 and determined to benot web, then the firewall rules are evaluated before forwarding on tothe Internet 504.

§ 12.0 Cloud IPS

FIG. 30 is a block diagram of a cloud IPS system 800, implemented viathe cloud system 500 and/or the distributed security system 100.Attackers are intruding into user host machines (i.e., the user device400, etc.) to gain server and data access. Static signatures will notwork as attackers are getting sophisticated. Further, SSL encryptedtraffic is an easy way for hackers to get away. Thus, the cloud IPSsystem 800 can provide complete protection on all ports and protocols,regardless of location, platform, operating system, etc. The cloud IPSsystem 800 protect users from Intrusion based on attack signatures,Known and unknown exploits; provides inline detection to identifyoutbound malware intrusion attacks and block/alert them; providesreal-time, granular control and visibility for globally unifiedadministration, policy management, and reporting; and provides SNORTbased signatures for detecting Command-and-Control (C2C), exploits, etc.on all ports and protocols.

In today's world, IPS is essential for security and is can be offered asa stand-alone solution, or incorporated in next-generation firewalls,the technology is more pervasive than ever before. But while mostcompanies have some form of IPS in place, there are questions about itseffectiveness. Due to the increase in user mobility and the skyrocketinguse of cloud services, users and apps have been leaving thenetwork-taking with them precious visibility, the key to IPS. As usersaccess applications off the network and often away from VPNs, they leavebehind the IPS, running blind.

Today's IPS solutions are primarily built for server protection, andattackers have moved to primarily targeting users. While protecting theserver still has its place, having an IPS that can follow the user andprovide always-on inspection of the user connection is fundamental tostopping today's intrusions. Traditional IPS approaches have difficultyscaling to meet the inspection demands of today's organizations. Addingto that challenge, a majority of threats now reside in SSL-encryptedtraffic, but there are limits to the amount of SSL traffic IPS hardwarecan inspect. As internet traffic and user demands increase,organizations must constantly balance the need for performance and theamount of traffic they can inspect. And they must often compromise byinspecting less, thereby increasing threat exposure and organizationalrisk.

Note, as described herein, the terms customer, enterprise, organization,etc. are used and all refer to some entity's network, computingresources, IT infrastructure, users, etc. Those skilled in the art willrecognize these different terms all relate to a similar construct,namely the enterprise 200, regional office 510, the headquarters 520,etc. That is, these terms collectively refer to computing and networkingresources that require protection by the cloud system 500 and/or thedistributed security system 100.

The objective of the cloud IPS system 800 is to provide an IPS service,via the cloud system 500 and/or the distributed security system 100. Bydelivering IPS from the cloud, all users and offices get always-on IPSthreat protection and coverage, regardless of connection type, platformtype, operating system, or location. The cloud IPS also restores fullvisibility into user, app, and internet connections, as all traffic onand off network is fully inspected. Because the cloud IPS 800 isdelivered as a service from the cloud system 500 and/or the distributedsecurity system 100, there is unlimited capacity to inspect usertraffic, even hard-to-inspect SSL traffic.

Most IPS solutions reside in the data center and lack the ability todeliver visibility and control to off-network traffic. These IPSsolutions lose more visibility and control every day, as mobility andSaaS take users and apps off the network. In addition, newerconnections-such as SD-WAN, 5G, and direct-to-internet-all encourageorganizations to embrace the Internet 504 as their corporate network.All these technological shifts have diminished the value of traditionalIPS and hindered its ability keep users safe.

Again, the cloud IPS system 800 delivers IPS from the cloud, whichallows IPS to follow the user, regardless of connection type orlocation. Every time a user connects to the Internet 504 or an app, thecloud IPS system 800 is there. It sits between the connection, inline,providing needed IPS threat visibility that traditional IPS solutionshave lost. Organizations can finally restore their lost visibility andthreat protection.

One of the major challenges facing traditional IPS solutions is theability to scale traffic inspection—and correctly sizing IPS solutionsis a real guessing game. What seems like the right size can quicklybecome insufficient as user demands grow, and that triggers costlyhardware refreshes. Even more challenging is the need for SSLinspection. The growth of SSL traffic is staggering—it has been reportedthat over 80 percent of enterprise traffic is now encrypted. But SSLinspection is performance intensive, which is why most IPS hardwaresolutions fall far short of the task. The result is organizations cannotinspect all their SSL traffic, and with a majority of threats now hidingin SSL, that's a serious risk.

The cloud IPS system 800 turns the inspection challenge into aneffortless afterthought. Because the cloud IPS system 800 is deliveredfrom the cloud system 500 and/or the distributed security system 100,inspection is elastically scaled based upon demand. Every user getsunlimited inspection capacity, so there is no need to guess how muchinspection is needed going forward. Best of all, SSL inspection isnative in the cloud system 500 and/or the distributed security system100, so there is the freedom to inspect all encrypted traffic.

There is a requirement to understand the meaning of alert data. Key tothis task is bringing in full user and application context from externalsources and correlating this data. However, many IPS solutions struggleto deliver because the meaningful context and correlation of threat datarequires thoughtful integration of multiple security systems. With thecloud IPS system 800, it is a fully integrated platform from day one,with no assembly required. Built from the ground up as a full securitystack delivered as a service, the cloud system 500 and/or thedistributed security system 100 provides multiple threat technologiesthat expertly work together to unify and correlate the threat data.Here, the cloud firewall 602, cloud sandbox, DLP, Cloud Access SecurityBroker (CASB), and web and content filtering are all integrated into aunified multi-tenant cloud service. Turn on the services needed, whenneeded, as demands grow. Because all relevant threat data is in oneplace, there is full user, file, and app context and the correlationneeded to understand the risk posture.

Also, maintaining an IPS can strain IT resources. Consistently testingand deploying IPS signatures is time consuming, error prone, and oftenrequires restrictive change windows. As a result, many companies fallbehind on updates, which increases risk. Delivered as a service, thecloud IPS system 800 is constantly updated transparently with the latestvulnerability coverage. Users will always get the latest threatprotection.

Further, the cloud system 500 and/or the distributed security system 100has millions of users and thousands of companies around the world,sharing threat data and intelligence. Thus, the cloud IPS system 800 isconstantly tracking emerging threats across the cloud and closelycollaborating with industry, military, and security organizations tokeep the cloud updated. As a result, one gets smarter threatintelligence designed to stop emerging threats quicker and reducecorporate risk. The cloud IPS system 800 can support tens of thousandsof signatures and continually update/add new signatures as required.

§ 13.0 Stream Scanning

The various functions described herein, the firewall 602, the cloud IPSsystem 800, etc., can utilize various approaches to scan traffic in thecloud node 502 and/or the processing node 110. Again, the followingdescription is presented with reference to the cloud node 502 in thecloud-based system 500, but those skilled in the art will recognize thesame applies to the processing node 110 in the distributed securitysystem 100. One scanning technique can be via a Security PatternMatching (SPM) engine that is block based typically used for web traffic(HTTP Header+Body scan) if the cloud node 502 proxy inspects HTTPtransactions.

Another approach includes a stream scanning approach described herein toperform a packet based or stream-based scan that matches rules based ona Snort style rule syntax. That is, the stream scanning approach borrowsthe Snort format for writing signatures. For example, the Snort formatcan be compliant to the SNORT User Manual 2.9.15.1, 2019, the contentsof which are incorporated by reference herein in their entirety.

A Snort rule can be broken down into 2 sections:

Rule Header Rule Options

The rule header specifies the connection parameters protocol, sourceIP/port and destination IP and port. The rule header is used to create atree for a rule database lookup. Here is the anatomy of the rule header.

Action Protocol Address Port Direction Address Port

For example, alert tcp any any->any 21

Various protocols are supported such as IP (matches any IP protocol),TCP, UDP, etc. Classless Inter-Domain Routing (CIDR) notation can beused to specify an IP address. Multiple IP address and ports can also bespecified. A few examples are given below:

alert tcp any any −> any [21,48,50] alert udp any any <− any :1024 alertip any any <> any 1024: alert tcp 192.168.0.0/16 any −> any :1024 alerttcp 192.168.0.0/16 any −> [192.168.0.0/16, 10.10.120.0/24, 172.11.12.41]any

The rule options follow the rule header and are enclosed inside a pairof parentheses. There may be one or more options. These options are thenseparated with a semicolon. If you use multiple options, these optionsform a logical AND.

For example, (msg:“DoS”; content:“server”; classtype:DoS;)

An example of a complete rule includes

alert tcp any any −> any 21 (msg:“DoS”; content:“server”;classtype:DoS;)

The following options are used to specify details about the threat.

Threatname is the threat name for the rule. The name for threats cancome from a threat library. New threats are found by a researcher andadded to the threat library.

Threatcat is an internal categorization of threats. This value can be aname or a number. Here are some example categories:

advthrt_advanced_security advthrt_suspicious_dest advthrt_phishingadvthrt_page_risk_ind advthrt_botnet advthrt_adspywareadvthrt_malware_site advthrt_webspam advthrt_peer_to_peeradvthrt_cryptomining advthrt_unauth_comm Advthrt_adspyware_sitesadvthrt_xss advthrt_exploit advthrt_browser_exploit advthrt_dos

Threatid is associated with the given threatname and is a uniqueidentifier for a threat in the threat library.

Experimental rules can be monitoring rules evaluated by a streamscanning (SS) engine normal like any other rule, but when these rulesare matched, then no policy is enforced by the cloud node 502. Theserules are however logged in the firewall/Weblog records for analysis.Here are two scenarios where these experimental rules will be helpful.

The best possible signature being written appears to be aggressive andmay result in False Positives (FPs). The guideline over here is to startwith monitor mode signature and then review the hits in production(actual use) for a time period for FPs before promoting the signature toblock mode category.

A threat campaign is being scoped out where a generic signature iswritten to gather domains/IPs which are then processed offline to getcontent and develop better block mode signatures. These experimentalsignatures do not result in block mode and are usually phased out aftera week or two.

Fast Pattern rules are applicable only to rules that have at-least onepattern specified using content or Perl Compatible Regular Expressions(PCRE) option. These are patterns that are evaluated first. A rule canexplicitly specify fast pattern by using the fast_pattern option. Themax fast pattern length can be 16 bytes. The fast pattern also should beat least 3 bytes long. A rule with PCRE option must contain at least onecontent which is eligible to be treated as fast pattern. If nofast_pattern is specified, then the longest content pattern is taken asfast pattern. A fast pattern is automatically taken as ONLY if it doesnot have any within or distance modifiers.

alert tcp any any −> any 80 (msg:″DOM Elements″;content:”document.getElementsByTagName”;)

For example, in the rule above since there is only one content, we willchoose the pattern specified by this content as the fast pattern.However, the pattern is larger than 16 bytes so we will only select 16bytes from the pattern. So “document.getElem” will be the fast pattern.The rule will be equivalent to writing.

alert tcp any any −> any 80 (msg:″DOM Elements″;content:”document.getElem”; fast_pattern:only;content:”document.getElementsByTagName”;)

Once the rules are written, then before loading up the rule file, itsbest to first validate the rules for any errors. A tool can be used tovalidate the rule file.

§ 13.1 Event Processing

The stream scanning supports rules with event processing using twooptions namely ‘detection_filter’ and ‘event_filter.’ The eventprocessing engine primarily uses the count of number of packets over acertain period of time. The count for the time period is maintained overmore than one stream/connection based on what the tracking attribute is.The tracking attribute can be either of source IP, source port,destination IP, and destination port. One key difference from Snort isthat Snort does not support tracking by ports. The timer for the countstarts after the first packet that matches the rule. Once the timerexpires, the new timer again starts after the first match of the rule.Therefore, it is important to note that the timer is NOT aligned to anyboundary (like second or hour) rather is dependent on the timing ofmatches. After the timer expires the counter is reset to 0.

Detection filters can be used to write rules that take intoconsideration the number of times the rule matches in a given timeperiod. Furthermore, the number of matches can be tracked based on IPaddress or port of source or the destination.

Detection_filter: track <by_src|by_dst|by_srcport|by_dstport), count c,seconds s;

An example of using a detection filter includes P

Alert tcp any any −> any 24 (msg:”example”; flags:S;detection_filter:track by_src, 20, 60;)

In the example above, the rule matches if the TCP packet is a SYN(Synchronize) packet (i.e., only the SYN flag is set). However, the rulematch alone does not generate an alert. The stream scanner will nowcheck if the detection_filter condition has also been satisfied, i.e.,if there are more than 20 SYN packets from the same source IP (as thecurrent packet's source IP) within 60 seconds then it will generatealert.

Note: All packets after the first packet that meets the condition of thedetection filter will generate alerts until the time period expires. Inthe example above after the 20th packet all packets will generate alertsfor that 60 second period. It is recommended to use event_filter toreduce the number of alerts.

Option Description track by_src | by_dst | Rate is tracked either by oneof these attribute. This means count is by_srcport | by_dstportmaintained for each unique source IP addresses or source port ordestination IP or destination port. count c number of rule matching in sseconds that will cause detection_filter limit to be exceeded. c must benonzero value. A value of −1 disables the detection filter seconds stime period over which count is accrued. s must be nonzero value.

Event filters can be used to reduce the number of alerts that aregenerated. If the rule contains both event_filter and detection_filter,the count and timer for both the options are calculated and maintainedseparately.

event_filter: type <limit|threshold|both>, track<by_src|by_dst|by_srcport|by_dstport), count c, seconds s;

Option Description type limit | threshold | both type limit alerts onthe 1st m events during the time interval, then ignores events for therest of the time interval. Type threshold alerts every m times we seethis event during the time interval. Type both alerts once per timeinterval after seeing m occurrences of the event, then ignores anyadditional events during the time interval. track by_src | by_dst | by_rate is tracked either by one of these attributes. This means srcport |by_dstport count is maintained for each unique source IP addresses orsource port or destination IP or destination port. count c number ofrule matching in s seconds that will cause event filter limit to beexceeded. c must be a nonzero value. A value of −1 disables the eventfilter seconds s time period over which count is accrued. s must benonzero value.

Example of using event_filter and detection_filter together. When usingtogether, event_filter only considers an event that has passed thedetection filter.

Alert tcp any any −> any 24 (msg:”example”; flags:S;detection_filter:track by_src, 3, 10; event_filter:type both, trackby_src, 2, 10;)

FIG. 31 is a diagram of detection filters and event filters usedtogether. In the example of FIG. 31, the event_filter will only alertonce on the 2nd alert for the 10 seconds period. Note that the start ofthe 10 seconds is from the first alert generated when the detectionfilter condition is true. Important point to note is that the timingperiod for detection_filter and event_filter is not the same. The timerfor detection_filter starts when the rule matches (i.e., in this casewhen SYN packet is seen) whereas the timer for the event_filter startswhen after the first alert is generated by the detection_filter.

§ 13.2 Differences from Snort

This section outlines what are the differences between the streamscanner and Snort when it comes to evaluating the rules. By default, thestream scanner will apply the offset and depth on each packet:

alert tcp any any->any 21 (content:“SMB”; offset:4; depth:5;)

If one wants the stream scanner to match a pattern over a stream ratherthan a single packet, the keyword ‘flow:stream’ can be used, i.e., theinternal tracking offsets by default always increases as more data isscanned (more packets are seen).

alert tcp any any −> any 21 (content:”SMB”; flow:stream; offset:4;depth:5;)

The above rule will only search for pattern between offset 4 to 9 in theentire stream irrespective of the number of packets seen. If the firstpacket is of size 10, then the search will terminate in the first packetitself.

The stream scanner does not buffer data. This is the most importantdifference with Snort. Internally, the stream scanner does not bufferany traffic data. This does not mean it will not maintain state forstreams. It can still apply pattern matching across packets boundary bymaintaining stream state.

The stream scanner can only scan up to the limit of max_scan_size. Thislimit can be overridden by specifying the max_scan_size in the rule. Forthe rules with flow:to_server or flow:from_server the limit is onlyapplied on the matching direction. For example, if the rule says‘flow:to_server’ then maximum is checked against only total bytes sentto the server. If the direction is not specified, then the rule appliesto total bytes in both directions.

alert tcp any any −> any 21 (msg:″DoS″; flow:to_server; content:”Test”;max_scan_size:100; classtype:DoS;)

In the above rule, the stream scanner will stop scanning if it sees morethan 100 bytes from the client side of the connection.

Apart from having max_scan_size, the stream scanner also enforces packetlimit. By default, the packet limit is 16 packets which includes packetsfrom both server and client side. This limit can be configured per ruleusing the option ‘max_count.’ The default value of Both max_count andmax_scan_size are enforced even if they are not specified in the rule.

alert tcp any any −> any 21 (msg:″DoS″; flow:to_server; content:”Test”;max_scan_size:100; max_count:10; classtype:DoS;)

In the above rule, the stream scanner will stop scanning if it sees morethan 10 packets (total of both directions) OR more than 100 bytes fromthe client side of the connection. The maximum value for max_count canbe the 2{circumflex over ( )}31, i.e., the limit for signed integer.

By default, Snort will continue matching further packets for flows evenafter a match is found. When using the stream scanner as part of thecloud node 502, it does not make sense to match more than a single rulefor a given session/transaction since it is only important to log onerule, and the action of (drop/allow) is based on the first matched rule.

alert tcp any any −> any 21 (msg:″DoS″; flow:to_server; content:”Test”;max_scan_size:100; single_match:yes; classtype:DoS;)

If you want to force a rule to do multiple match in the cloud node 502,you need to set “single_match:no” explicitly.

alert tcp any any −> any 21 (msg:″DoS″; flow:to_server; content:”Test”;max_scan_size:100; single_match; classtype:DoS;)

If yes and no is not specified then “yes” is assumed. For example in theabove rule, it is the same as specifying “single_match:yes.” This onlyapplies to fast pattern on patterns that are larger than 16.

Let's look at two different cases with below 2 rules.

alert tcp any any −> any 21 (msg:″DoS″; content : ”for_firstof_part-other_part”; content: ″example1″; classtype:DoS;) alert tcp any any −>any 21 (msg:″DoS″; content:”example2”;content:”for_firstof_part-other_part”; fast_pattern; classtype:DoS;)

In the above rules, fast patterns are longer than 16 characters/bytes.In the first rule, the first content is automatically taken as fastpattern while in the second rule the second content is manuallyspecified as the first pattern. In both cases, the first 16 charactersof the pattern are chosen and the rules is rewritten internally asbelow:

alert tcp any any −> any 21 (msg:″DoS″; content:”for_firstof_part”;fast_pattern:only; content:”for_firstof_part-other_part”; content:″example1″; classtype:DoS;) alert tcp any any −> any 21 (msg:″DoS″;content:”for_firstof_part”; fast_pattern:only; content:”example2”;content:”for_firstof_part-other_part”; classtype:DoS;)

Now for the First rule, if the fast patterns falls across two packetboundaries then the above rules will never match.

Packet 1 Packet 2 a sample data for_first_of_ part-other_part example1

We would have matched the derived fast pattern (that was broken fromoriginal pattern) across the pattern boundary at packet 2. But when wego to match the actual complete pattern maximum, we will rewind is onlyup to the beginning of the Packet 2 where it will fail to find thecomplete match. The fix for the above rule is to dedicate a pattern forfast_pattern and manually specify it as fast_pattern:only. This way thestream scanner will not break the pattern into 2 parts internally.

alert tcp any any −> any 21 (msg:″DoS″; content:”for_firstof_part-other_part”; fast_pattern:only; content: ″example1″; classtype:DoS;)

Similarly, for the second rule, we will only start pattern matching ifthe fast pattern matches first. The fast pattern will only match inpacket 2. However when we go to match the content:“example 2,” wealready missed it in packet 1. Hence this rule will not get triggered.

Packet 1 Packet 2 a example2 data for_first_of_ part-other_part

When multiple patterns are specified, the stream scanner can recursivelygo back if a pattern fails to match. For example, in the rule below.Recursion can originate from content or byte test options

alert tcp any any −> any 21 (msg:″DoS″; content:”this”; content:”is”;distance:0; within:3; content:”ok”; within:”3”;)

And for the data—Let's say first this is not ok and later this is ok.

The first ‘this is’ in the data will match the first 2 content optionsof the rule, however, the “not” will disqualify the 3rd content frommatching. When the 3rd content fails to match, we will go back to againto match the second content “is” from the last matched point. When even“is” fails, then we go all the way back to first content “this” to matchfrom the last matched point (i.e., after the first ‘is’ in the data).Recursion can slow the process of matching rules and impact performance.

If any dsize is present, then it is the first option to be evaluated,irrespective of the location of dsize within the rule. The dsize willonly consider the current size of the packet. Note that dsize is notapplied on the size of the stream. Dsize may not be reliable due to TCPreassembly or fragmentation. If dsize fails then the entire packet isskipped.

alert tcp any any −> any 21 (msg:″DoS″; content:”master”; dsize:14;content: ″server″; classtype:DoS;)

Will test for dsize first. The content pattern will only be searched onpackets that are EXACTLY 14 bytes.

alert tcp any any −> any 21 (msg:″DoS″; content:”master”; dsize:>14;)

Will search for pattern “master” on all TCP packets to destination port21 that have packet size greater than 14.

The stream scanner uses two different versions of regex (RegularExpression) engines to evaluate PCRE patterns.

§ 13.3 Supported Options

Here is a list of example options that are supported by the streamscanner:

Option Description content Test against a string pattern pcre Testagainst a regex pattern. Support for pcre flags is limited. Please referto the secion ‘Supported PCRE flags’ byte_test Extract bytes and testagainst a value. ‘dce’ is not supported byte_extract Extract bytes andstore. ‘dce’ is not supported. Extracted variable cannot be used acrossrules. byte_jump Jump to a offset based on extracted bytes. ‘dce’ is notsupported isdataat Check the size of the payload distance Modifier forcontent/pcre within Modifier for content/pcre offset Modifier forcontent/pcre depth Modifier for content/pcre flags Filters on TCP flags.Supports ignoring of flags too. flow Supported options To_server &from_client From_server & to_client No_stream Established (alwaysenabled) flowbits Supported actions 1. set 2. unset 3. isset 4. Isnotset5. noalert nocase Caseless match for content, pcre fast_pattern Sets thepattern as fast_pattern stream_size Limit the size of the streamdetection_filter Post detection option based on rate of packetevent_filter Controls the amount of alerts for each rule. Only allowedwithin rule and not as a standalone option.

Here is list of options, that are added to support specific use cases inthe cloud system 500 and to integrate with other modules such as thefirewall 602, the cloud IPS system 800, etc.

Option Description threatname Threat name from threat database threatidThreat id threatcat Threat Category (same was Malware category in Weblogs) rank Threat Rank fw_appid Firewall Application Id (for future use)experimental Marks the rule as experimental. Has effects only when usingSS in inline mode in the cloud node noalert Same as flowbits: noalert;Does all evaluation of the rule but does not generate an alert when rulematches. In API mode, the rule will not generate a callback. max_scan_Same as stream_size but limit applied on both directions size oftraffic.

These rules do not impact the evaluation of the signature and is addedonly to make external rules usable in the stream scanner.

Option Description msg Message to display. classtype The snort classtypereference Reference for the rule metadata Metadata for the rule sidSnort id gen_id rev The revision number

Supported HTTP Modifiers for Content

Option Description http_uri The URI for http http_method The HTTP Methodhttp_header Start of HTTP header http_cookie The HTTP cookiehttp_client_body The start of HTTP client body

Example of using HTTP modifiers for content are given below:

alert tcp any any −> any any 80 (msg:″example website″;content:”www.example.com”; http_uri;) alert tcp any any −> any any 80(msg:″example website″; content:”POST”; http_method; distance:2;content:”www.example.com”; http_uri; )

§ 13.4 Rule Evaluation Mechanism

This section describes the internal workings of the stream scanningengine. It gives an overview of how the internal structures are createdfor rules and what sequence is followed to evaluate rules. The designprinciple closely tries to mimic the Snort evaluation order while alsokeeping performance in mind.

§ 13.4.1 Rule Grouping

During the compilation stage, similar rules are grouped together basedon criteria (described in the next topic). These groups are referred toas lookup objects. The grouping step constructs a lookup tree whoseedges are individual lookup objects (rule groups). During the data path,the stream scanner will first walk through the lookup tree and selectone or more lookup objects to evaluate. Once lookup objects areselected, the stream scanner will proceed to match fast patterns for therules in those lookup groups. FIG. 32 is a diagram illustrating rulegrouping in a lookup tree.

§ 13.4.2 Grouping Attributes

Below are the attributes used to create the lookup tree. Each level ofthe lookup tree indicates a grouping attribute. For example, at thetop-level protocol is used as a grouping attribute. At the root node ofthe lookup tree each branch represents a unique value of protocol.

Algorithm Used for Order Attribute Description Searching 1 Protocol TCP,UDP, ICMP or any Linear Table 2 Destination The destination port fromTCP and Linear Table Port UDP Datagram. For ICMP it's always 0(signifies any) 3 Destination IP IPv4 Destination IP from the IPPatricia Tree with bit widths datagram [4, 4, 4, 4, 4, 4, 4, 4] 4 SourcePort The source port from TCP and UDP Patricia Tree with bit widthdatagram. For ICMP its always 0. [3, 3, 3, 3, 4] 5 Source IP IPv4 SourceIP from the IP Datagram. Patricia Tree with bit widths [4, 4, 4, 4, 4,4, 4, 4] 6 Direction The direction of Traffic flow. Linear TableDetermined by to_server (flow: to_server) or from server (low:from_server) or both if no direction is specified.

§ 13.4.3 Fast Pattern on Lookups

The fast pattern of rules belonging to the same lookup object (rulegroup) are added into single multi pattern search interface. Searchinterfaces are lower-level string search engines like aho-corasick,hyperscan or sregex. Currently these fast patterns are added to aaho-corasick search engine (acism). There could be one or more matchesamong the fast patterns. The stream scanner then takes the individualrules that belong to the matched fast pattern and starts evaluatingthose rules. Some Rules in the rule group may not have any pattern(neither contents nor PCRE) at all. These rules are called default rulessince they will immediately be evaluated once a lookup is selectedbefore fast pattern evaluation is performed. A group can containmultiple default rules along side with multiple rules with patterns.

§ 13.4.4 Single Rule Evaluation

All rules selected from the lookup objects are evaluated individually.For rules with patterns, evaluation only starts if the fast patternmatches. A rule can contain multiple rule options (like content,byte_test, etc.). These options form what is called a rule option DAG(directed acyclic graph). For a rule to match all the rule options haveto match. FIG. 33 is a diagram of an example rule option DirectedAcyclic Graph (DAG).

The dependency in the DAG is determined by the rule option modifiers.For example, assume two ‘content’ options and the second ‘content’ has a‘within’ modifier. The second ‘content’ has a dependency on the first‘content’ and is only evaluated if the first contain matched. Similarly,if there are options that use a variable defined by ‘byte_extract,’ thenall those options will be dependent on the byte_extract option. Thoseoptions will only be evaluated after byte_extract succeeds.

During pattern matching, if recursion occurs then, we may walk back inDAG. The dotted lines in FIG. 33 represent the recursion transition forthe states. For example, if ‘pattern2’ fails in the above DAG then itwill clear the state of ‘pattern1’ from matched to non-matched. In theexample above, note that ‘pcre’ is dependent on ‘content:“pattern3”’because of the ‘R’ (Relative) flag.

§ 13.4.5 from Data to Matches

FIG. 34 is a diagram of the overall flow when data arrives on the streamscanning engine. At first, we extract the grouping attributes from thedata, then using the attribute we walk through the lookup tree andselect the matching lookup objects. It might find none, one or manylookup objects. If no lookup object is found for given attributes, thenthe search terminates there.

Now it processes the lookup objects individually. Rules that do not haveany fast pattern are immediately evaluated. For other rules with fastpattern first the fast pattern must match before it is evaluated. Oncethe fast pattern is matched, the rule is evaluated individually. Therule options DAG is evaluated, and if all the options matches, then therule is finally matched. The stream scanning engine can match multiplerules for a given data.

§ 14.0 Stream Scanner on a Cloud Node

The cloud node 502 (or the processing node 110) can execute a streamscanning engine. The stream scanning engine runs in the firewall NATlayer hence all traffic destined for NAT'ing is scanned. Note thatnon-NAT'd traffic like HTTP, HTTPS does not go through the streamscanning engine. Any bridged traffic, i.e., non-HTTP traffic bridgedusing HTTP protocol, is also passed through the stream scanning engine.FIG. 35 is a flowchart of scan processing at a cloud node 502, with aStream Scanning Engine (SSE) and Security Pattern Matching (SPM).

§ 14.1 Threat Detection Between SPM and SSE

Threats policy can be specified both in Proxy configuration(Admin>policy>Advanced Threat Protection) and from IPS rules(Admin>policy>IPS Control). There can be a situation where Proxyconfiguration for Advanced Threat Protection is set to Allow; however,IPS rules for the same threat category is set to Block/Drop. Atransaction is identified to contain a threat but is allowed by theProxy because the action is set to allow, However Firewall will blockthe session containing this Transaction. The end result is that the saidtransaction will still be blocked.

FIG. 36 is a flow diagram of functions performed by the cloud node 502between a firewall module and a proxy module.

When the cloud node 502 starts up for the first time, it loads the rulefile specified by the option ‘rulefile’ in sc.conf. A new rule file canbe loaded on a running cloud node 502. Any files pushed through smcdssgets loaded and replaces the current database.

In production deployment for privacy and security reasons, we cannotkeep plain text rules files. Specially in private and virtual cloudnodes 502 where customers have login access, it is important they not beable to access the rules. To enable privacy of rule files, the streamscanner supports reading encrypted files. The stream scanner can bedisabled by not providing any rule file.

§ 15.0 Stream Scanner Statistics

The stream scanner maintains various statistics internally that can beused to measure the performance of a particular rule (including rulegroups) or a pattern. These stats are internally always collected.

It will be appreciated that some embodiments described herein mayinclude one or more generic or specialized processors (“one or moreprocessors”) such as microprocessors, digital signal processors,customized processors, and Field-Programmable Gate Arrays (FPGAs) andunique stored program instructions (including both software andfirmware) that control the one or more processors to implement, inconjunction with certain non-processor circuits, some, most, or all ofthe functions of the methods and/or systems described herein.Alternatively, some or all functions may be implemented by a statemachine that has no stored program instructions, or in one or moreApplication-Specific Integrated Circuits (ASICs), in which each functionor some combinations of certain of the functions are implemented ascustom logic. Of course, a combination of the aforementioned approachesmay be used. Moreover, some embodiments may be implemented as anon-transitory computer-readable storage medium having computer-readablecode stored thereon for programming a computer, server, appliance,device, etc. each of which may include a processor to perform methods asdescribed and claimed herein. Examples of such computer-readable storagemediums include, but are not limited to, a hard disk, an optical storagedevice, a magnetic storage device, a ROM (Read Only Memory), a PROM(Programmable Read-Only Memory), an EPROM (Erasable ProgrammableRead-Only Memory), an EEPROM (Electrically Erasable ProgrammableRead-Only Memory), Flash memory, and the like. When stored in thenon-transitory computer-readable medium, the software can includeinstructions executable by a processor that, in response to suchexecution, cause a processor or any other circuitry to perform a set ofoperations, steps, methods, processes, algorithms, etc.

Although the present disclosure has been illustrated and describedherein with reference to preferred embodiments and specific examplesthereof, it will be readily apparent to those of ordinary skill in theart that other embodiments and examples may perform similar functionsand/or achieve like results. All such equivalent embodiments andexamples are within the spirit and scope of the present disclosure, arecontemplated thereby, and are intended to be covered by the followingclaims.

What is claimed is:
 1. A non-transitory computer-readable storage mediumhaving computer readable code stored thereon for programming aprocessor, in a node of a cloud-based security system, to perform stepsof: obtaining a plurality of rules each define via a rule syntax thatincludes a rule header and rule options, wherein each rule header isused to for a rule database lookup, and each rule options is used tospecify details about the associated rule; monitoring data associatedwith a user of the cloud-based security system; analyzing the data withthe plurality of rules; and performing one or more security functions onthe data based on triggering of a rule of the plurality of rules.
 2. Thenon-transitory computer-readable storage medium of claim 1, wherein oneor more rules of the plurality of rules include experimental rules, andwherein the steps further include monitoring the data associated withtriggering any of the experimental rules; and processing domainsassociated with the triggering any of the experimental rules to furtherdevelop the experimental rules.
 3. The non-transitory computer-readablestorage medium of claim 2, wherein the experimental rules are written ina manner that includes false positives, and the processing is to reducethe false positives.
 4. The non-transitory computer-readable storagemedium of claim 1, wherein the plurality of rules include a Snort-likeformat.
 5. The non-transitory computer-readable storage medium of claim1, wherein the node does not buffer the data during the monitoring, andwherein the steps further include maintaining a stream state acrosspacket boundaries of the data.
 6. The non-transitory computer-readablestorage medium of claim 1, wherein the steps further include subsequentto a first match of the rule of the plurality of rules, stopping theanalyzing and performing the one or more security functions.
 7. Thenon-transitory computer-readable storage medium of claim 1, wherein oneor more rules of the plurality of rules include a fast pattern.
 8. Thenon-transitory computer-readable storage medium of claim 1, wherein thecloud-based security system utilizes the plurality of rules to implementone or more of a firewall and an intrusion prevention system.
 9. A nodein a cloud-based security system, comprising: a processor and memorystoring instructions that, when executed, cause the processor to obtaina plurality of rules each define via a rule syntax that includes a ruleheader and rule options, wherein each rule header is used to for a ruledatabase lookup, and each rule options is used to specify details aboutthe associated rule; monitor data associated with a user of thecloud-based security system; analyzing the data with the plurality ofrules; and performing one or more security functions on the data basedon triggering of a rule of the plurality of rules.
 10. The node of claim9, wherein one or more rules of the plurality of rules includeexperimental rules, and wherein the instructions that, when executed,further cause the processor to monitor the data associated withtriggering any of the experimental rules; and process domains associatedwith the triggering any of the experimental rules to further develop theexperimental rules.
 11. The node of claim 9, wherein the plurality ofrules include a Snort-like format.
 12. The node of claim 9, wherein thenode does not buffer the data during the monitoring, and wherein theinstructions that, when executed, further cause the processor tomaintaining a stream state across packet boundaries of the data.
 13. Thenode of claim 9, wherein the instructions that, when executed, furthercause the processor to subsequent to a first match of the rule of theplurality of rules, stopping the analyzing and performing the one ormore security functions.
 14. The node of claim 9, wherein one or morerules of the plurality of rules include a fast pattern.
 15. The node ofclaim 9, wherein the cloud-based security system utilizes the pluralityof rules to implement one or more of a firewall and an intrusionprevention system.
 16. A method implemented in a node in a cloud-basedsecurity system comprising obtaining a plurality of rules each definevia a rule syntax that includes a rule header and rule options, whereineach rule header is used to for a rule database lookup, and each ruleoptions is used to specify details about the associated rule; monitoringdata associated with a user of the cloud-based security system;analyzing the data with the plurality of rules; and performing one ormore security functions on the data based on triggering of a rule of theplurality of rules.
 17. The method of claim 16, wherein one or morerules of the plurality of rules include experimental rules, and furthercomprising monitoring the data associated with triggering any of theexperimental rules; and processing domains associated with thetriggering any of the experimental rules to further develop theexperimental rules.
 18. The method of claim 16, wherein the node doesnot buffer the data during the monitoring, and further comprisingmaintaining a stream state across packet boundaries of the data.
 19. Themethod of claim 16, further comprising subsequent to a first match ofthe rule of the plurality of rules, stopping the analyzing andperforming the one or more security functions.
 20. The method of claim16, wherein the cloud-based security system utilizes the plurality ofrules to implement one or more of a firewall and an intrusion preventionsystem.